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Abstract 

Deriving  weights  for  a  Value  Focused  Thinking  (VFT)  hierarchy  demands 
considerable  time  and  input  from  Decision  Makers  (DM)  and  Subject  Matter  Experts 
(SME).  Often,  the  DMs  and  SMEs  are  the  leaders  of  companies  and  organizations,  and 
this  required  time  is  unrealistic  with  their  schedules.  In  these  situations,  as  well  as 
scenarios  where  the  are  no  available  DMs  /  SMEs,  conventional  means  of  weighting  a 
VFT  hierarchy  are  impossible,  and  any  VFT  analysis  is  halted.  When  historical  data 
exists  on  evaluation  measures  and  performance  of  alternatives,  linear  programming  and 
genetic  algorithm  based  optimization  may  be  used  to  derive  historically  optimal  weights 
for  a  hierarchy.  Analysis  may  then  be  done  to  determine  the  utility  of  transposing  these 
weights  into  a  hierarchy  to  evaluate  a  current  list  of  alternatives.  This  type  of  analysis  is 
also  useful  in  “first  cut”  weighting  of  a  hierarchy,  and  therefore  reduces  the  time 
demands  for  DMs/SMEs  to  complete  the  weighting  process.  This  methodology  can 
provide  insight  into  any  situation  where  historical  information  exists  on  ordinally  ranked, 
competing  alternatives. 
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LINEAR  PROGAMMING  AND  GENETIC  ALGORITHM  BASED  OPTIMIZATION 


FOR  THE  WEIGHTING  SCHEME  OF  A  VALUE  FOCUSED  THINKING 

HIERARCHY 


Chapter  1.  Introduction 


“If  a  man  will  begin  with  certainties  he  will  end  with  doubts,  but  if  he  will  be  content 
to  begin  with  doubts  he  shall  end  in  certainties”  (Francis  Bacon) 


1.0  Background 

Decision  analysis  provides  effective  methods  for  organizing  a  complex 
problem  into  a  structure  that  can  be  analyzed.  In  particular,  elements  of  a 
decision’s  structure  include  the  possible  courses  of  action,  the  possible 
outcomes  that  could  result,  the  likelihood  of  those  outcomes,  and  the 
eventual  consequences  (e.g.,  costs  and  benefits)  to  be  derived  from  the 
different  outcomes.  (Clemen  1996) 

Decision  analysis’  influence  has  been  rapidly  spreading  as  decision-makers  are 
looking  for  a  better  way  to  analyze  what  is  truly  important  in  a  decision.  Although  its 
origins  date  back  to  the  mid-sixties,  when  Ronald  Howard  defined  the  field  while  he  was 
an  Associate  Professor  at  MIT,  the  popularity  for  decision  analysis  has  mainly  grown 
over  recent  years.  Brainstorming  and  other  conventional  methods  of  attacking  hard 
decisions  make  it  nearly  impossible  to  arrive  at  a  quality  decision  without  forgetting 
major  aspects  of  the  objective,  not  considering  highly  qualified  alternatives,  and  allowing 
personal  bias  to  completely  dominate  the  decision.  The  educated,  formal  logic  of 
decision  analysis  appeals  to  the  decision-maker  (DM)  for  its  thoroughness  and  strategic 
logic  (Clemen  1996).  It  has  been  used  by  countless  companies  to  better  their  ways  of 
business,  and  done  so  successfully  in  many  fields  of  study  (Decision  1999). 
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Value  focused  thinking  (VFT),  a  subset  of  decision  analysis,  deals  with  multiple- 
objective  decision-making.  The  objectives  are  competing  and  require  consideration  of 
tradeoffs  among  these  objectives  (Kirkwood  1997).  These  multiple,  competing 
objectives  are  the  outline  for  what  a  decision-maker  “values”  in  their  decision.  The  main 
objectives  that  go  into  the  decision  are  broken  down  into  sub-objectives,  and  those  sub¬ 
objectives  are  further  broken  into  their  own  sub-objectives,  until  the  sub-objectives  can 
be  quantified  into  “measures”  which  directly  or  indirectly  give  a  rating  to  each  value. 

VFT  is  implemented  using  a  ten  step  iterative  process,  which  will  be  discussed  in  depth 
within  chapter  two. 

VFT’s  goal  is  to  uncover  what  is  truly  valued  in  a  fundamental  objective  (issue  of 
importance).  After  the  values  are  broken  down  in  a  hierarchical  fashion,  the  different 
facets  of  this  decision  are  “weighted”  by  relative  importance.  Values  with  higher  weights 
often  drive  the  decision,  and  all  values  are  tradeoffs  in  that  the  sum  of  the  weights  under 
each  value  equals  one.  The  completed  VFT  process  allows  one  to  rank  alternatives 
depending  on  how  high  of  a  score  they  receive  by  the  hierarchy. 

Assigning  weights  to  the  different  values  under  a  hierarchy  is  often  a  topic  of 
concern  (Kirkwood  1997).  Group  discussions,  meetings  with  company  leaders,  surveys 
of  employees,  etc,  are  examples  of  ways  to  determine  a  hierarchy’s  weights.  This  input 
from  professionals  in  the  field  of  study  is  a  critical  step  in  determining  the  weights  of  the 
hierarchy.  They  are  considered  the  decision  makers  (DM)  and/or  subject  matter  experts 
(SME)  in  the  area  of  interest.  These  professionals  are  the  people  that  do  the  work,  day  in 
and  day  out,  and  know  the  system  the  best.  Without  their  input,  conventional  VFT 
weighting  techniques  are  simply  not  possible.  Being  without  their  input  is  not 
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uncommon.  In  the  military,  getting  hours  of  a  General’s  time  is  highly  unlikely.  If  a 
hierarchy  is  built  with  the  objective  of  determining  what  an  enemy  or  competitor  values, 
there  is  little  chance  they  will  weight  the  hierarchy.  Historically,  this  type  of  roadblock 
ended  VFT  possibilities. 

In  cases  where  historical  data  is  readily  available,  statistical  analysis  and 
optimization  may  provide  an  acceptable  means  of  determining  the  weights  when 
professionals  are  unavailable.  With  a  strong  hierarchy  based  on  measures  that  may  be 
determined  through  the  historical  data,  and  knowledge  of  past  performance  of  competing 
alternatives,  one  may  solve  for  the  weights  of  the  hierarchy  which  best  match  up  with  the 
known  past  perfonnance.  The  derived  weights  give  a  projected  ranking,  which  if 
optimal,  will  mirror  the  true  historical  rankings.  Once  these  weights  are  determined, 
analysis  may  be  done  on  current  alternatives,  by  projecting  the  previous  weights  into  the 
future  (current  time  period). 

The  optimization  needed  to  solve  for  these  weights  is  based  on  minimizing  an 
error  function  that  is  subject  to  constraints  that  all  sub-categorical  weights  for  the  values 
and  measures  in  the  hierarchy,  sum  to  one.  The  error  function  may  vary  widely 
depending  on  the  desired  outcome,  but  it  is  based  on  the  difference  between  the  known 
ranking  of  the  alternatives  and  the  projected  rankings  of  these  alternatives  after  the 
weights  for  the  hierarchy  are  detennined. 

The  hybrid  Evolutionary/Classical  Solver  within  Premium  Solver®  is  used  to 
optimize  the  weighting  scheme  of  the  hierarchy.  “The  hybrid  Evolutionary/Classical 
Solver  uses  genetic  algorithm  methods  such  as  mutation,  crossover,  selection  and 
constraint  repair,  but  also  uses  deterministic,  gradient-free  direct  search  methods, 
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classical  gradient-based  quasi-Newton  methods,  and  even  the  Simplex  method  for  linear 
subsets  of  the  constraints”  (Premium  1). 

1.1  Problem  Statement 

The  purpose  of  this  thesis  is  to  provide  a  methodology  to  weight  a  Value  Focused 
Thinking  hierarchy  when  input  from  professionals  is  unavailable.  Excel’s 
Evolutionary/Classical  Solver®  is  used  to  optimize  the  weights  of  a  hierarchy  by 
minimizing  an  error  function  that  is  based  on  matching  historical  rankings  of  competing 
alternatives. 

1.2  Research  Objective 

This  thesis  integrates  optimization  with  value  focused  thinking  to  solve  for  the 
weights  of  a  VFT  hierarchy.  If  historical  data  and  performance  are  available,  and  what  is 
truly  valued  in  a  fundamental  objective  is  consistent,  the  knowledge  of  the  past  can  be 
projected  into  the  future  through  solving  for  the  weights  of  the  hierarchy.  By  this  means 
of  weighting  being  successful,  the  benefits  of  the  Value  Focused  Thinking  process 
increases  dramatically.  The  VFT  process  no  longer  ends  when  the  SMEs  or  DMs  are 
unable  to  weight  the  hierarchy.  This  process  may  also  provide  useful  “first-cut”  insight 
into  the  weighting,  even  if  the  SMEs  and/or  DMs  are  available,  or  in  a  situation  where 
one  attempts  to  gain  insight  into  an  “enemy’s”  process.  The  time  constraints  on  these 
individuals  can  be  greatly  reduced  by  using  this  methodology  to  solve  for  the  weights, 
thus  providing  a  superior  starting  point  for  the  DM’s/SME’s  input. 
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1.3  Methodology 

The  first  step  of  this  research  is  to  select  an  example  to  explore  the  weighting 
technique  used  to  estimate  the  weights  during  the  use  of  Value  Focused  Thinking.  This 
example  must  contain  the  necessary  components  in  order  for  this  weighting  scheme  to  be 
possible: 

1 :  Enough  historical  data/infonnation  must  exist  in  order  to  build  a  hierarchy 
with  the  measures  of  that  hierarchy  being  based  upon  this  data/information. 

2:  This  data/infonnation  must  be  available  on  multiple  competing  “alternatives”. 

3:  In  relation  to  one  another,  the  historical  perfonnance  of  these  alternatives  must 
be  known. 

Once  this  example  is  selected,  a  hierarchy  must  be  built  around  the  fundamental 
objective.  Next,  an  error  function  must  be  formulated  with  the  desired  goals  of  the 
optimization.  The  weights  of  the  hierarchy  are  determined  through  optimization  of  the 
historical  data  inputs  for  the  measures  of  the  hierarchy.  The  weights  may  be  solved  for 
different  time  periods  and  durations.  With  a  best-suited  weighting  scheme  for  the  current 
period,  each  alternative  will  be  scored  under  this  hierarchy,  and  the  resulting  “choice” 
will  be  known.  The  weighting  of  the  value  focused  thinking  process  is  now  complete, 
and  done  without  any  input  from  a  decision  maker  or  subject  matter  expert. 

1.4  Thesis  Outline 

The  literature  review  will  look  at  relevant  literature  that  provides  a  necessary 
background  to  understand  the  methodology  of  this  thesis.  Through  examples,  chapter 
three  will  illustrate  the  methodology  behind  using  optimization  to  solve  for  the  weights  of 
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a  VFT  hierarchy.  This  will  include  a  mathematical  formulation  for  the  optimization,  and 
a  7-step  process  that  walks  through  the  major  aspects  of  this  methodology.  Chapter  four 
will  provide  a  real  world  example  of  using  the  proposed  methodology.  The  example  will 
look  at  a  hierarchy  with  the  fundamental  objective  of  “what  does  one  value  in  a  military 
construction  (MILCON)  project.”  The  weights  of  this  hierarchy  will  be  optimized  using 
historical  data,  and  the  resulting  weights  will  provide  insight  into  what  the  decision¬ 
makers  historically  valued.  Finally  chapter  five  will  summarize  the  strengths, 
weaknesses,  and  insight  gained  from  this  thesis.  It  will  also  list  possible  areas  for  future 
study  on  this  topic. 
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Chapter  2.  Literature  Review 


2.0  Introduction 

The  purpose  of  this  chapter  is  to  review  relevant  literature  and  provide  a 
fundamental  background  in  the  areas  of  Decision  Analysis,  Value  Focused  Thinking,  and 
Linear  Programming.  This  chapter  will  begin  with  an  in-depth  discussion  on  Decision 
Analysis,  followed  by  an  introduction  into  a  subset  of  Decision  Analysis,  Value  Focused 
Thinking,  and  finally  give  a  short  background  into  one  of  the  most  popular  optimization 
methods,  Linear  Programming.  Through  studying  this  chapter,  the  reader  should  have  an 
adequate  background  to  understand  the  methodology  introduced  in  chapter  3. 

2.1  Decision  Analysis 

What  is  a  decision?  The  one  essential  element  of  a  decision  is  the 
existence  of  alternatives.  That  is,  you  must  have  choice  to  make  between 
at  least  two  different  things,  only  one  of  which  you  can  select.  If  you 
don’t  have  alternatives,  then  you  may  have  a  problem,  but  it  isn’t  a 
decision  problem.  (Of  course,  some  of  the  most  vexing  decisions  are 
those  where  you  do  not  seem  to  have  any  good  alternatives.)  (Kirkwood 
1997) 

The  field  of  decision  analysis  (DA)  was  defined  by  Ronald  Howard  in  1964  while 
he  was  Associate  Professor  of  Electrical  Engineering,  Associate  Professor  of  Industrial 
Management,  and  Associate  Director  of  the  Operations  Research  Center  at  MIT 
(Howard).  He  joined  the  Stanford  Faculty  in  1965  and  has  been  teaching  decision 
analysis  thereafter.  Since  1964,  decision  analysis  has  been  a  rapidly  growing  field. 

Many  consulting  companies  are  using  a  variation  of  the  process,  and  numerous  Fortune 
500  companies  are  currently  setting  up  their  own  decision  analysis  consulting  groups 
(Decision  Education  2000).  Numerous  professional  societies  associated  with  decision 
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analysis  have  arisen,  such  as  the  Decision  Analysis  Affinity  Group,  a  society  in  which 
over  fifty  major  companies  participate,  and  INFORMS,  a  predominantly  operations 
research  society. 

Decision  analysis  is  a  set  of  quantitative  methods  for  analyzing  decisions  based  on 
the  axioms  of  consistent  choice  (Keefer  2000).  It  is  a  structured  and  strategic  way  of 
thinking  about  how  the  action  taken  in  the  current  decision  would  lead  to  a  result 
(Spradlin  1997).  DA  helps  to  eliminate  much  of  the  bias  that  often  hinders  decision 
makers.  Clemen  explains  that  there  are  four  basic  sources  of  difficulty,  necessitating  the 
use  of  decision  analysis  to  simplify  the  decision-making  process  (2).  The  first  source  of 
difficulty  is  complexity.  Often  times  there  are  many  different  facets  to  the  decision  at 
hand.  Keeping  track  of  these  issues  and  accurately  defending  each  of  their  positions  in 
ones  mind  is  nearly  impossible.  Inherent  uncertainty  in  the  situation  is  the  second  source 
of  difficulty. 


For  example,  imagine  a  firm  trying  to  decide  whether  to  introduce  a  new 
product.  The  size  of  the  market,  the  market  price,  eventual  competition, 
and  manufacturing  and  distribution  costs  all  may  be  uncertain  to  some 
extent,  and  all  have  some  impact  on  the  firm’s  eventual  payoff.  A 
decision-analysis  approach  can  help  in  identifying  important  sources  of 
uncertainty  and  representing  that  uncertainty  in  a  systematic  and  useful 
way.  (Clemen  1996) 

The  third  source  of  difficulty  deals  with  multiple  objectives  in  which  progress  in  one 
direction  may  impede  progress  in  another.  This  is  where  a  decision  maker  is  forced  to 
trade  off  a  benefit  in  one  area  with  a  cost  in  another.  Clemen’s  final  source  of  difficulty 
arises  if  different  perspectives  lead  to  different  conclusions,  or  even  if  a  single 
perspective  with  slight  changes  to  inputs,  leads  to  a  different  choice. 
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This  source  of  difficulty  is  particularly  pertinent  when  more  than  one 
person  is  involved  in  making  the  decision.  Different  individuals  may  look 
at  the  problem  from  different  perspectives,  or  they  may  disagree  on  the 
uncertainty  or  value  of  the  various  outcomes.  The  use  of  decision-analysis 
framework  and  tools  can  help  sort  through  and  resolve  these  differences 
whether  the  decision  maker  is  an  individual  or  a  group  of  stakeholders 
with  diverse  opinions.  (Clemen  1996) 

Clemen  breaks  down  the  decision  analysis  process  into  the  following  flowchart: 


Figure  2.1:  Decision  Analysis  Process  Flowchart 

Although  this  is  Clemen’s  DA  flowchart,  it  holds  close  to  the  form  of  many  author’s 
stepwise  progression  through  the  decision  analysis  process.  For  this  reason,  his  flowchart 
will  be  used  to  depict  the  DA  process  in  general. 
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Identify  the  decision  situation  and  understand  objectives: 

The  first  step  of  the  decision  analysis  process,  identifying  the  decision  situation 
and  understanding  the  objectives,  is  a  critical  process.  “Although  we  usually  do  not  have 
trouble  finding  decisions  to  make  or  problems  to  solve,  we  do  sometimes  have  trouble 
identifying  the  exact  problem,  and  thus  we  sometimes  treat  the  wrong  problem”  (Clemen 
1996).  Understanding  the  objectives  is  important  in  that  the  objectives  need  to  be  on  the 
front  of  the  decision  makers  mind  in  order  to  keep  them  from  straying  off  what  is 
important.  “A  primary  difficulty  in  many  decisions  is  figuring  out  what  things  are 
important  in  evaluating  the  consequences  of  a  decision”  (Kirkwood  1997). 

Identify  alternatives: 

Identifying  alternatives  is  the  next  step  in  the  decision  analysis  process  in  which 
alternatives  that  are  intuitively  obvious,  and  those  less  obvious  alternatives  which  meet 
the  criteria  of  the  fundamental  objective,  make  up  a  list  of  possible  alternatives. 

Kirkwood  noted  the  complexity  of  this  task  when  he  stated  “Another  difficulty  in 
decision  making  is  detennining  the  relative  importance  of  different  aspects  of  the 
consequences.”  (2).  Careful  examination  of  the  objectives  should  lead  to  alternatives  that 
would  not  have  been  otherwise  considered.  Although  the  number  of  alternatives  must  be 
limited  to  a  reasonable  number,  it  is  important  to  have  a  diverse  list  of  objectives  that 
stretch  over  broad  horizons.  Kirkwood  warns  that  “Many  people  do  no  cast  their  nets 
wide  enough  when  they  consider  alternatives.”  (Kirkwood  1997).  He  also  notes  that  the 
complexity  of  many  modem  management  decisions  is  at  fault  for  many  of  the  alternatives 
not  being  considered.  Kirkwood  finally  warns  against  relying  solely  on  specialists,  for 
they  often  analyze  options  they  know  how  to  solve,  rather  than  the  preferable  ones. 
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Decompose  and  model  the  problem  /  Choosing  the  best  alternative: 

Decomposing  and  modeling  the  problem,  along  with  choosing  the  best  alternative, 
is  what  Clemen  and  many  other  authors  classify  as  the  “modeling  and  solution”  of  the 
problem  (7).  These  two  steps  are  the  heart  of  the  decision  analysis  process,  and  where 
the  majority  of  time  and  effort  is  spent.  Clemen  breaks  down  the  “Decompose  and  model 
the  problem”  into  modeling  of  the  problem  structure,  modeling  of  uncertainty,  and 
modeling  of  preferences.  The  goal  of  the  decomposition  of  the  problem  is  to  simplify  the 
structures  and  measure  value  and  uncertainty.  By  breaking  down  the  problem  into  more 
definable  elements,  the  decision  maker  may  in  turn  gain  more  insight  into  the  individual 
elements  and  better  model  each  facet  of  the  overall  problem.  Decision  trees  coupled  with 
probabilistic  uncertainty  models  are  the  foundation  for  the  “modeling”  process  as  a 
whole.  In  value  focused  thinking,  which  will  later  be  explained  at  length,  a  hierarchical 
model  is  introduced  in  which  multiple  competing  objectives  are  simultaneously 
considered.  The  accuracy  of  ones  model  directly  reflect  the  quality  of  insight  obtained 
through  the  decision  analysis  process. 

Sensitivity  Analysis: 

Sensitivity  analysis  is  the  arena  in  which  one  can  check  the  robustness  of  their 
model.  It  can  be  used  to  determine  the  impact  on  the  ranking  of  alternatives,  given  some 
change  in  various  model  assumptions  (Kirkwood  1997).  If  the  optimal  decision  changes 
with  the  minor  altering  of  the  model,  the  decision  is  considered  sensitive  and  the  decision 
maker  is  encouraged  to  reconsider  those  particular  aspects  of  the  model  (Clemen  1996). 
As  shown  in  Clemen’s  flowchart,  sensitivity  analysis  can  send  one  back  to  any  of  the  first 
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three  steps  of  the  DA  process,  including  defining  the  objectives.  It  is  very  common  to 
follow  this  cyclic  pattern  Clemen’s  flow  chart  depicts  until  the  objectives  are  clearly 
defined,  most  all  alternatives  are  considered,  and  models  are  refined.  This  is  the  common 
process  of  decision  analysis. 

2.2  Value  Focused  Thinking 

Developed  by  Keeney  in  1992,  Value  Focused  Thinking  (VFT)  is  decision 
analysis  that  deals  with  multi-objective  decision-making  (Keeney  1998).  VFT  is  a  great 
tool  that  has  been  successfully  implemented  in  a  vast  number  of  professional  arenas  and 
studies,  including  the  Operational  Analysis  of  Air  Force  2025  (Jackson  1996),  deciding 
new  policies  (Keeney  1998),  selecting  and  implementing  security  procedures  for 
transporting  nuclear  waste  (Keeney  1998),  and  selecting  construction  sites  for  critical 
installations  (Keeney  1993).  As  Ralph  Keeney  explains,  VFT  consists  of  two  activities: 
deciding  what  you  want  and  figuring  out  how  to  get  it  (4).  This  way  of  thinking,  Keeney 
claims,  gets  one  closer  to  their  goal  than  conventional  alternative  focused  thinking  in 
tenns  of  “here  are  my  options,  which  is  the  best.”  Keeney  illustrates  many  of  VFT’s 
benefits  with  the  following  diagram: 
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Figure  2.2:  Benefits  of  Value  Focused  Thinking 


2.2.1  The  Ten  Step  Process 

Chambal  uses  a  10-step  VFT  flowchart,  similar  to  Clemen’s  DA  flowchart,  which 
methodically  outlines  the  value  focused  thinking  process  (Class  2002): 
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Figure  2.3:  The  Ten  Step  Process 


Although  this  ten-step  process  appears  sequential  in  nature,  it  is  important  to  note  that  it 
is  truly  iterative.  Keep  in  mind  that  many  iterations  of  the  following  “steps”,  may  be 
taken  before  the  process  is  complete. 

Step  1:  Problem  Identification 

Problem  identification  is  the  step  in  the  VFT  process  where  the  groundwork  is 
laid  for  the  rest  of  the  analysis.  This  is  where  the  brainstorming  of  the  actual  problem 
takes  place,  and  where  the  question  “what  is  our  ultimate  goal”,  is  answered.  This  is  also 
the  milestone  where  the  fundamental  objective  is  solidified.  Keeney  defines  an  objective 
as  “. .  .something  that  one  desires  to  achieve”  and  characterizes  it  by  three  features:  a 
decision  context,  an  object,  and  a  direction  of  preference  (Keeney  1992).  He  also  stresses 
that: 

Values  of  decision-makers  are  made  explicit  with  objectives.  Hence,  the 
set  of  objectives  developed  for  a  decision  frame  is  absolutely  critical.  The 
fundamental  objectives  are  the  basis  for  any  interest  in  the  decision  being 
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considered.  There  objectives  qualitatively  state  all  the  is  of  concern  in  the 
decision  context.  They  also  provide  guidance  for  action  and  the 
foundation  for  any  quantitative  modeling  or  analyses  that  may  follow  this 
qualitative  articulation  of  values.  (Keeney  1992) 

For  the  illustrative  purposes  of  this  chapter,  the  problem  of  “Where  should  I  go  to 

college”  will  be  examined.  It  should  be  noted  that  this  example’s  purpose  is  to  give  a 

basic  understanding,  and  the  rigor  of  this  example  is  far  less  than  would  be  taken  in  a  true 

VFT  process. 

Step  2:  Create  Value  Hierarchy 

Creating  the  value  hierarchy  is  likely  the  most  difficult  step  in  the  VFT  process. 
Kirkwood  describes  a  value  hierarchy  as  a  value  structure  with  a  hierarchical  or 
“treelike”  structure  (Kirkwood  1997).  It  is  the  model  that  breaks  down  a  fundamental 
objective  into  smaller  sub-components,  and  finally  into  measurable  units.  The  reason 
why  this  step  is  so  difficult  is  because  of  the  complexity  of  determining  “what  you  truly 
value.”  An  example  of  this  would  be  the  question  of  “where  should  I  go  to  college”. 

The  following  chart  is  the  beginning  of  a  possible  hierarchy  built  for  this  fundamental 
objective: 


Figure  2.4:  College  Hierarchy 
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In  the  example,  the  three  pillars  determining  “where  should  I  go  to  college”  are 
academics,  location,  and  student  life.  These  values  could  be  broken  down  further,  for 
example  “what  do  I  value  in  academics,  etc. . or  measures  could  be  determined  to  rate 
the  individual  performance  of  a  college  under  each  of  the  values.  It  is  critically  important 
to  ensure  that  values  under  a  fundamental  objective  are  independent.  That  is,  making 
sure  that  the  preference  for  the  level  of  one  evaluation  measure  does  not  depend  on  the 
level  of  the  other  evaluation  measure  (Kirkwood  1997).  Kirkwood  lists  independence 
along  with  completeness,  nonredundancy,  operability,  and  small  size,  all  as  desirable 
properties  of  a  value  hierarchy  (16).  While  building  the  value  hierarchy,  the  question  of 
money  typically  arises.  In  many  value  hierarchies,  such  as  the  “where  should  I  go  to 
college”  hierarchy,  intuition  suggests  that  the  cost  would  play  a  major  role.  Money  or 
cost,  however,  will  not  be  included  in  the  value  hierarchy.  It  is  recommended  to  leave 
money  out  of  the  hierarchy,  and  only  use  a  benefit/cost  ratio  based  on  the  overall  benefit 
(Kirkwood  1997). 

Step  3:  Develop  evaluation  measures 

An  evaluation  measure  as  explained  by  Kirkwood  is: 

A  measuring  scale  for  the  degree  of  attainment  of  an  objective  is  an 
evaluation  measure.  Thus,  “annual  salary  in  dollars”  might  be  the 
evaluation  measure  for  the  job  seeker’s  objective  of  finding  a  higher 
salary.  Other  terms  that  are  sometimes  used  for  an  evaluation  measure  of 
effectiveness,  attribute,  perfonnance  measure,  or  metric.  (12) 

Evaluation  measures  are  how  an  individual  alternative  is  scored  on  how  well  it  meets  a 

value.  For  example,  the  case  of  the  college  hierarchy  may  have  three  measures,  one  for 

each  sub-objective: 
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Figure  2.5:  College  Hierarchy  with  Measures 

In  the  example,  the  quality  of  academics  is  “measured”  by  the  average  high  school  grade 
point  average  of  the  students  attending  the  college.  The  quality  of  location  is  “measured” 
by  whether  the  school  is  in  the  north,  south,  east,  or  west.  Finally,  student  life  would  be 
“measured”  by  the  number  of  student  organizations  present  on  the  college  campus. 
“Evaluation  measure  scales  can  be  classified  as  either  natural  or  constructed,  and  also 
either  direct  of  proxy”  (Kirkwood  1997).  Therefore,  the  four  types  of  measures  are 
natural-direct,  constructed-direct,  natural-proxy,  and  constructed-proxy. 

Kirkwood  explains  the  different  scales  as: 

Natural-  In  general  use  with  a  common  interpretation  by  everyone 
Constructed-  Developed  for  a  particular  decision  problem  to  measure  the  degree 
of  attainment  of  an  objective 

Direct-  Directly  measures  the  degree  of  attainment  of  an  objective 
Proxy-  Indirectly  reflects  the  degree  of  attainment  of  its  associated  objective 

Step  4:  Create  value  functions 

Single  dimension  value  functions  (SDVFs)  are  a  way  of  placing  all  of  the 
measures  on  a  common  scale,  industry  standard  for  which,  is  0  to  1  (0  being  worst,  and  1 
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being  best).  SDVFs  translate  a  score  relative  to  the  individual  measure  into  this  zero  to 
one  scale.  To  illustrate  the  most  frequently  used  SDVFs,  consider  the  example  of 
“average  high  school  GPA”  measure  under  the  college  hierarchy: 


Constant 


Figure  2.6:  Constant  returns  to  scale 

As  you  can  see  from  Figure  2.6,  as  in  all  the  value  functions,  the  higher  the  average  high 
school  GPA,  the  better.  Because  the  function  is  linear,  with  each  increment  in  the  x-axis 
(GPA),  the  increase  in  value  stays  constant.  This  is  called  constant  returns  to  scale. 


Decreasing 


Figure  2.7:  Decreasing  returns  to  scale 
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The  concave  nature  of  figure  2.7  represents  decreasing  returns  to  scale.  This  is  the 
situation  where  a  lot  of  value  is  added  early  on  for  an  increase  in  average  GPA,  however 
as  GPA  increases,  the  increase  in  value  lessens. 


Figure  2.8:  Increasing  returns  to  scale 

The  convex  nature  of  figure  2.8  represents  increasing  returns  to  scale.  This  is  the 
scenario  where  less  value  is  added  early  on  for  an  increase  in  average  GPA,  however  as 
GPA  increases,  the  increase  in  value  rises  with  each  increment  of  the  x-axis. 
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Figure  2.9:  Other  returns  to  scale,  ie.  S-curve 

Other  curves,  such  as  the  s-curve  illustrated  in  figure  2.9,  may  also  be  used  as  single 
dimension  value  functions.  The  s-curve  is  ideal  when  little  value  is  gained  early  on  or 
late  for  per  increase  in  GPA,  however  the  most  gain  in  value  is  found  during  some  middle 
period.  The  value  gained  per  increase  in  GPA  slowly  grows  in  early  times,  until  some 
maximum  slope  is  reached,  then  this  delta  gradually  decreases  for  the  remainder  of  the  x- 
axis. 
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Figure  2.10:  Piecewise  linear  single  dimension  value  functions 

Piecewise  linear  single  dimension  value  functions  have  constant  returns  to  scale  within 
each  region.  These  regions  are  discrete  increments  of  the  x-axis,  where  the  slope  differs 
from  the  surrounding  periods.  In  the  graph  above,  although  constant  in  the  individual 
intervals,  the  increase  in  value  per  increase  in  GPA  is  different  from  0-1.5,  as  it  is  from 
1. 5-2.5,  2. 5-3.5,  and  3.5-4. 

As  is  apparent  from  the  previous  examples,  these  single  dimension  value 
functions  translate  an  abstract  measurement,  such  as  average  high  school  GPA,  into  a 
quantifiable  zero  to  one  value  score. 


Step  5:  Weight  the  value  hierarchy 

Weights  provide  tradeoffs  between  competing  objectives. 

From  these  properties  of  the  single  dimensional  value  functions,  it  follows 
that  the  weight  for  an  evaluation  measure  is  equal  to  the  increment  in 
value  that  is  received  from  moving  the  score  on  that  evaluation  measure 
from  its  least  preferred  level  to  its  most  preferred  level.  This  property 
provides  a  basis  for  a  procedure  to  detennine  the  weights.  (Kirkwood 
1997) 
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There  are  multiple  ways  to  weight  a  hierarchy,  all  based  on  preference.  Kirkwood 
discusses  “swing  weighting”,  others  prefer  a  direct  method,  but  all  methods  have  the 
same  goal  of  determining  preference  among  competing  objectives.  The  importance  is  in 
having  a  comfortable  allocation  of  the  weights  across  the  values  in  the  hierarchy.  In  the 
college  hierarchy,  possible  weighting  could  be  as  follows: 


In  this  example  academics  are  most  valued,  therefore  they  are  given  a  relatively  high 
weight  of  .6,  compared  to  the  weights  of  .2  and  .2  for  location  and  student  life 
respectively.  The  weighting  of  location  and  student  life  show  that  these  are  equally 
valued  in  deciding  “where  should  I  go  to  college.” 

Step  6:  Alternative  generation 

Alternative  generation  is  very  similar  to  the  step  in  the  general  DA  process  where 
the  goal  is  to  get  a  wide  variety  of  candidates  to  be  evaluated  by  the  hierarchy. 
Alternative  generation  is  the  step  in  the  VFT  process  where  all  possible  alternatives 
which  meet  the  criteria  of  the  fundamental  objective  need  to  be  considered.  From  these 
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possibilities,  a  diverse  list  of  alternatives  needs  to  be  selected.  Careful  examination  of  the 
objectives  should  lead  to  alternatives  that  would  not  have  been  otherwise  considered. 
Although  the  number  of  alternatives  must  be  limited  to  a  reasonable  number,  it  is 
important  to  have  a  diverse  list  of  objectives  that  stretch  over  broad  horizons.  In  the 
college  example,  it  would  be  desirable  to  have  schools  from  all  over  the  country,  with 
different  levels  of  academics  and  student  life  (Harvard,  University  of  Florida,  Notre 
Dame,  Air  Force  Academy,  Berkeley,  etc. . .). 

Step  7:  Alternative  scoring 

The  process  of  scoring  alternatives  determines  how  an  alternative  scores  on  the 
“x-axis”.  The  raw  scores  are  found  for  each  of  the  measures.  For  example,  in  the  case  of 
the  college  hierarchy,  the  average  high  school  GPA  of  the  Air  Force  Academy  may  be  a 
3.8.  This  is  the  score  along  the  x-axis  of  the  SDVF,  for  this  particular  measure.  This 
process  is  done  for  every  measure,  for  each  of  the  alternatives 

Steps  8:  Deterministic  Analysis 

The  step  of  deterministic  analysis  is  where  an  alternate’s  overall  value  is 
calculated.  The  value  of  an  alternative  is  calculated  in  the  following  manner: 
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v(x)  =  x  VVV  (x  ) 
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Zw=l 


The  multiobjective  value  function 


Tj-  (-T- )  —  The  single  dimensional  value  function  / 
W-  =  The  weight  for  evaluation  measure  i 


Figure  2.11:  The  Value  Function 

In  words,  the  multi-objective  value  function  is  equal  to  the  sum  of  all  “scored”  single 
dimension  value  function  values  multiplied  by  their  respective  evaluation  measure 
weight.  These  weights,  w;,  are  what  capture  the  relative  importance  of  an  individual 
measure.  Once  all  alternates  receive  an  overall  score,  these  scores  may  be  compared  to 
create  a  ranked  list. 

Steps  9:  Sensitivity  Analysis 

This  step  is  where  the  “what  if’  questions  of  the  analysis  are  answered,  and  the 
sensitivity  of  the  decision  is  determined.  Determining  the  sensitivity  involves 
manipulating  weights  to  see  where  rankings  change.  If  the  outcome  is  not  sensitive  to 
change,  weights  and  SDVFs  may  be  moderately  changed,  without  any  change  in  the 
ranking  of  the  top  alternate.  If  the  outcome  is  sensitive  however,  these  slight  changes 
affect  this  ranking,  and  the  confidence  in  the  top  alternate  lessens.  This  sensitivity 
analysis  shows  how  truly  close  competing  alternatives  are. 
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Steps  10:  Conclusions  and  Recommendations 


This  step  provides  insight  and  summarizes  the  findings  through  the  ten-step  VFT 
process.  A  final  recommendation  is  made  regarding  the  fundamental  objective,  keeping 
in  mind  the  sensitivity  and  costs  of  this  recommendation. 

2.3  Problems  Without  a  Decision  Makers 

It  is  not  uncommon  to  be  faced  with  a  Value  Focused  Thinking  problem  in  which 
no  “decision  maker(s)”  are  available  for  the  weighting  of  the  hierarchy.  This  is  often  a 
problem  because  either  the  decision  makers  are  too  busy,  or  they  are  simply  unavailable 
to  provide  input.  As  mentioned  earlier,  this  problem  may  also  arise  if  the  VFT  process  is 
attempting  to  gain  insight  into  a  competing  organization  or  enemy.  If  the  decision 
makers  are  not  present,  conventional  means  of  weighting  the  hierarchy  are  eliminated. 

The  scenario  is  especially  prevalent  in  the  military.  General  officers’  schedules 
rarely  allow  room  for  frequent  meetings  and  discussions  focused  on  providing  feedback 
into  “what  they  truly  value”  in  a  hierarchy.  The  available  time  may  be  limited,  therefore 
any  insight  into  a  first  cut  of  the  weighting  would  greatly  benefit  the  process.  The 
General  may  be  the  only  individual  who  can  accurately  weight  the  hierarchy,  but  a  first 
cut  will  provide  a  starting  point  for  solicitation  These  are  the  times,  in  the  military  as 
well  as  the  civilian  world,  where  another  means  of  weighting  the  hierarchy  must  be 
explored. 

2.4  Using  Linear  Programming  to  Select  Appropriate  Weights 

This  study  proposes  using  linear  programming  as  a  solution  to  determining 
appropriate  weights  when  decision  makers  are  unavailable.  If  a  past  performance  of 
alternatives  is  known  and  if  any  type  of  common  rating  criteria  is  available  for  the 
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alternatives,  linear  programming  may  be  used  to  determine  the  weights.  The  derived 
weights  will  minimize  the  difference  between  the  known  and  predicted  rankings.  This 
assumes  that  the  past  known  rankings  are  available. 

2.5  Linear  Programming 

George  B.  Dantzig  is  credited  with  the  introduction  of  linear  programming  after 
his  development  of  the  simplex  method  in  1947  (Bazarraa  xi).  It  was  developed  during 
and  after  World  War  II,  due  to  the  necessity  for  planning  and  coordination  among  various 
projects  and  the  need  for  efficient  utilization  of  scarce  resources  (Bazaraa  vii).  The 
modeling  capabilities  linear  programming  provides  has  made  it  a  success  in  many  fields 
of  study. 


Since  the  development  of  the  simplex  method  many  people  have 
contributed  to  the  growth  of  linear  programming  by  developing  its 
mathematical  theory,  devising  efficient  computational  methods  and  codes, 
exploring  new  algorithms  and  new  applications,  and  by  their  use  of  linear 
programming  as  an  aiding  tool  for  solving  more  complex  problems,  for 
instance,  discrete  programs,  nonlinear  programs,  combinatorial  problems, 
stochastic  programming  problems,  and  problems  of  optimal  control. 
(Bazarraa  vii) 

The  development  of  linear  programming  has  been  ranked  among  the  most  important 
scientific  advances  of  the  mid-20th  century  (Hillier  25).  The  most  often  seen  linear 
programming  problem  involves  allocating  limited  resources  among  competing  activities 
in  an  optimal  way.  This  is  optimizing  a  function,  while  satisfying  linear  constraints.  In 
the  following  definitions,  Bazarraa  breaks  linear  programming  down  in  a  mathematical 
manner  (2)  (Bazarraa  gives  a  minimization  example,  with  >  constrains.  Maximization 
problems,  and  both  >  and  <  constraints  are  also  possible): 

Consider  the  following  linear  programming  problem: 
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Minimize  z  =  C1X1  +  C2X2  +  . . .  +  cnxn 


Subject  to:  anxi  +  ai2X2  +  ....  +  ainxn  >  bi 
a2lXl  +  a22X2  +  . . . .  +  a2nxn  >  b2 

amiXi  +  am2X2  +  . . . .  +  amnXn  >  bm 
xi,x2,....,xn>  0 

Here  cixi  +  C2X2  +  . . .  +  cnxn  is  the  objective  function  (or  criterion  function)  to  be 
minimized  and  will  be  denoted  by  z.  The  coefficients  ci,  C2,. . . .,  cn  are  the  (known)  cost 
coefficients  and  xt,  X2,. . . .,  xn  are  the  decision  variables  to  be  determined.  The 
inequalities  denote  the  constraints  (or  restrictions).  The  coefficients  aq  for  i  =1,2,. . .,m, 
and  j=l ,2, . . .,n  are  called  the  technological  coefficients.  These  technological  coefficients 
form  the  constraint  matrix  A. 

l12  -•  O 

l22  ""  a2n 

1  -  ....  a  1 
m2  mn ) 

The  column  vector  whose  ith  component  is  bj,  which  is  referred  to  as  the  right-hand-side 
vector,  represents  the  minimal  requirements  to  be  satisfied.  The  constraints  xi,  X2,  . . .,  xn 
>  0  are  the  non-negativity  constraints.  A  set  of  variables  xi,  . . .,  xn  satisfying  all  the 
constraints  is  called  a  feasible  point  or  a  feasible  vector.  The  set  of  all  such  point 
constitutes  the  feasible  region  or  feasible  space. 

Using  the  foregoing  terminology,  the  linear  programming  problem  can  be  stated 
as  follows:  Among  all  feasible  vectors,  find  one  that  minimizes  (or  maximizes)  the 


A  := 


all  2 


a21  2 


a  ,  a 
^  ml 
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objective  function  (Bazarra  2).  Chapter  3  will  incorporate  this  methodology  into  a 
general  technique  to  solve  the  weights  of  a  value  focused  thinking  model  with  historical 
data  present. 

2.6  Summary 

This  chapter  has  provided  a  review  of  Decision  Analysis,  Value  Focused 
Thinking,  and  optimization.  These  tools  will  prove  critical  in  the  following  chapter, 
where  the  methodology  behind  using  optimization  to  select  the  weights  of  a  VFT 
hierarchy  will  be  explored. 
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Chapter  3.  Methodology 


“A  problem  well  stated  is  a  problem  half-solved.”  (Charles  F.  Kettering) 

3.0  Overview 

This  chapter  illustrates  the  methodology  behind  using  non-linear  programming 
and  genetic  algorithms  to  optimize  the  weights  of  a  VFT  hierarchy.  This  methodology 
will  be  explained  both  in  a  numerical  and  theoretical  manner  in  order  to  highlight  the 
important  facets  of  using  this  type  of  weighting  scheme.  The  chapter  begins  with  a 
purely  mathematical  and  expandable  example,  depicting  all  major  aspects  of  the 
methodology.  This  will  be  followed  by  a  simple  example  using  the  defined  tenninology. 
Finally,  a  larger,  multi-time-period  example  will  be  shown,  completing  the  explanation  of 
this  methodology. 

The  methodology  of  this  chapter  will  be  conducted  in  a  seven-step  process: 

1)  Build  the  VFT  hierarchy,  including  measures  and  SDVFs. 

2)  Obtain  the  historical  data  pertaining  to  each  of  the  measures  and  the  rankings 
for  each  time  period  of  interest. 

3)  Determine  appropriate  error  function,  subject  to  VFT  constraints. 

4)  Formulate  the  optimization  to  minimize  the  error  function. 

5)  Solve  for  the  optimal  weights  of  the  hierarchy. 

6)  Transpose  these  weights  into  the  hierarchy. 

7)  Use  the  hierarchy  to  predict  the  rankings  of  the  next  time  period  of  interest. 
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3.1  Mathematical  Representation  of  Methodology 

This  section  will  focus  on  the  mathematical  representation  of  the  methodology 
utilized  to  optimize  the  weights  of  a  VFT  hierarchy.  The  section  begins  with  breaking 
down  the  VFT  hierarchy  into  its  elements,  followed  by  defining  additional  terminology 
pertinent  to  this  chapter.  Finally,  an  expandable,  mathematical  formulation  of  the 
methodology  will  be  explained. 

The  following  charts  and  definitions  depict  the  individual  elements  of  a  hierarchy 
(note  that  hierarchies  may  be  small  or  large  with  a  varying  number  of  tiers  and/or 
branches): 

Fundamental  Objective:  The  overall  question  one  desires  to  answer.  The  purpose  of  the 
VFT  process  is  to  gain  insight  into  this  fundamental  objective. 

Tier  of  the  Hierarchy:  A  tier  of  the  hierarchy  is  the  common  horizontal  position  of  sub¬ 
objectives  or  measures  within  the  hierarchy.  There  are  tradeoffs  between  any  sub¬ 
objectives  within  a  tier  as  long  as  they  fall  under  a  common  objective. 
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Figure  3.1:  First  Tier  of  Hierarchy 


Figure  3.2:  Second  Tier  of  Hierarchy 


Figure  3.3:  Third  Tier  of  Hierarchy  (measures  in  this  example) 
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Branch  of  the  Hierarchy:  A  branch  of  a  hierarchy  is  all  of  the  sub-objectives  down  to 
measures  of  the  hierarchy,  which  stem  from  one  of  the  first  tier  sub-objectives. 


Figure  3.4:  First  Branch  of  Hierarchy 


Figure  3.5:  Second  Branch  of  Hierarchy 
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Figure  3.6:  Third  Branch  of  Hierarchy 

In  attempt  to  give  a  clear,  mathematical  fonnulation  of  the  methodology,  consider 
the  following  7-step  process  of  implementing  the  methodology.  First,  consider  the 
following,  expandable  hierarchy: 
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Step  1:  Build  Hierarchy 


Figure  3.7:  Expandable  Hierarchy 

Definitions  for  the  expandable  hierarchy  are  as  follows: 

Vij  =  value  in  the  hierarchy. 

i  =  tier  of  the  hierarchy,  i  =  0,1,2...  (t-1) 

j  =  value’s  ordered  placement  within  the  ith  tier  of  the  hierarchy, 
t  =  final  tier  of  the  hierarchy,  therefore  t-1  represents  the  last  tier  of  values. 

Mj  =  measure  in  the  hierarchy. 

j  =  measure’s  ordered  placement  within  the  final  tier  (tth  tier)  of  the  hierarchy. 
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j  =  1,2... k 

k  =  total  number  of  measures  in  the  hierarchy. 

Wij  =  local  weight  assigned  to  each  value  (Vy  )  of  the  hierarchy. 

Wtj  =  local  weight  assigned  to  each  of  the  “k”  measures  (Mj )  of  the  hierarchy, 
t  =  final  tier  of  the  hierarchy. 

j  =  measure’s  ordered  placement  within  the  final  tier  (t’th  tier)  of  the  hierarchy, 
j  =  1,2... k 

Wtjg=  global  weight  of  measure  j.  j  =  1,2 _ k 

S;  =  the  overall  score  of  alternative  i.  i  =  1,2. .  .N. 

Vj(xj)  =  y-axis  value  of  measure  j,  as  translated  by  the  SDVF,  for  one  alternate. 

Dy  =  the  set  of  all  the  descendants  of  Vij.  A  set  of  descendants  are  all  of  the  values  or 
measures  which  “descend”  from  a  common  parent  node  (where  the  parent  node  is  located 
on  tier  i- 1 .  The  sum  of  the  weights  of  these  descendants  must  sum  to  1 :  ( 

t  . .  s  Aj  =  l ) 

V=1 

v  =  ordered  placement  within  a  set  of  descendants, 
k  =  number  of  values  or  measures  in  the  descendant  set. 

Step  2:  Data 

In  order  to  complete  this  process,  historical  data  on  each  measure,  and 
performance  of  each  alternative  is  needed.  The  following  table  lists  the  y-axis  “values” 
of  the  measures  after  they  have  been  translated  into  the  0-1  scale  by  the  SDVFs. 

Time  period  1  _ _ _ _ _ 


Alt  1 

Alt  2 

Alt  3 

***** 

Alt  N 

Mo 

Vo(xo) 

Vo(xo) 

Vo(xo) 

***** 

Vo(xo) 

Mi 

Vi  (xi) 

Vi  (xi) 

Vi  (xi) 

***** 

Vi(xi) 
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M2 

V2(X2) 

V2(X2) 

V2(X2) 

***** 

V2(X2) 

Mk 

Vk(xk) 

Vk(xk) 

Vk(xk) 

***** 

Vk(xk) 

Time  period  2 


Alt  1 

Alt  2 

Alt  3 

***** 

AltN 

MO 

Vo(xo) 

Vo(xo) 

Vo(xo) 

***** 

Vo(xo) 

Ml 

Vi  (xi) 

Vi  (xi) 

Vi  (xi) 

***** 

Vi(xi) 

M2 

V2(X2) 

V2(X2) 

V2(X2) 

***** 

V2(X2) 

Mk 

Vk(xk) 

Vk(xk) 

Vk(xk) 

***** 

Vk(xk) 

Time  period  f 


Alt  1 

Alt  2 

Alt  3 

***** 

AltN 

Mo 

Vo(xo) 

Vo(xo) 

Vo(xo) 

***** 

Vo(xo) 

Mi 

Vi  (xi) 

Vi  (xi) 

Vi  (xi) 

***** 

Vi(xi) 

M2 

V2(X2) 

V2(X2) 

V2(X2) 

***** 

V2(X2) 

Mk 

Vk(xk) 

Vk(xk) 

Vk(xk) 

***** 

Vk(xk) 

Figure  3.8:  Historical  Data 

For  each  time  period  also  exists  known  rankings,  Rank;/ ,  for  each  alternative: 


Period  1 

Period  2 

***** 

Period  f 

Alt  1 

Ranknk 

Rank12k 

***** 

Ranklfk 

Alt  2 

Rank21k 

Rank22k 

***** 

Rank2fk 

Alt  3 

Rank31k 

Rank32k 

***** 

Rank3fk 

AltN 

RankN1k 

RankN2k 

***** 

RankNfk 

Figure  3.9:  Historical  Rankings 
Step  3:  Error  Function 

period/  AltN 

E  =  ^  Rank*  -  Rank /  \ 

j=period\  i=Alt\ 
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Rank;/  is  the  known  rank  of  alternative  i  in  time  period  j,  and  Rank;/  is  the  calculated 
rank  of  alternative  i  in  time  period  j.  Once  again,  it  is  important  to  note  that  this  is  not  the 
only  error  function  that  may  be  used. 

Step  4:  Formulate  Optimization 

In  order  to  formulate  the  optimization,  it  is  first  necessary  to  compute  RankijC. 
RankijC  is  calculated  by  rank  ordering  the  overall  scores  of  all  N  alternatives.  In 
Microsoft  Excel,  this  is  done  with  the  command:  RANK(Scorei,  ScoreN).  The  overall 
score  of  an  individual  alternative,  Si,  is  calculated  by  summing  the  global  weights  of  all 
of  the  measures  multiplied  by  their  respective  Vj(xj).  The  global  weight  of  a  measure, 

Wtjg,  is  found  by  taking  the  product  of  the  chain  of  the  parent  nodes’  local  weights  (up  to 
and  including  the  objective  function),  and  multiplying  that  value  by  the  local  weight  of 
that  measure,  Wtj: 

Obj  Function 

W/=W4*  I  I  W, 

ij=Wy 's  parent  node 

Therefore,  alternative  i’s  overall  score  is  calculated: 

j= o 

In  turn  the  calculated  ranking  of  alternative  i  in  period  j,  RankyC  is  the  ranking  of  S, 
among  all  N  alternatives,  in  period  j.  With  this  notation  defined,  a  mathematical 
representation  of  the  optimization  is  as  follows: 

period  f  AltN 

Minimize  E  =  I  Ei  Rankj  -  Rank  y  | 

j = period  1  i=Alt\ 

Subject  to:  ^  RE  =  1  V  W(M)  r  e  Dy  (the  sum  of  all  local  weights  =  1,  i  =  1 . .  .t-1) 
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(all  weights  must  be  between  zero  and  one) 


0  <  W,  <  1 

ij 

Step  5,  6,  &  7: 

Using  Excel’s  Premium  Solver,  solve  for  the  optimal  weights  of  the  hierarchy. 
Integrate  these  optimized  weights  into  the  original  hierarchy.  With  a  complete  hierarchy, 
one  may  now  score  a  current  list  of  alternatives,  and  predict  the  ordinal  outcome  of  these 
alternatives. 

3.2  Example  1 

The  following  examples  in  this  chapter  are  not  real-world  scenarios,  with 
fictitious  hierarchies  and  data.  Consider  the  following  example,  conducted  using  the  7- 
step  process. 

Step  1:  Build  Hierarchy 


Figure  3.10:  Hierarchy  Ex.  1 
Step  2:  Obtain  Historical  Data 
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Consider  the  following  data,  created  solely  for  the  purpose  of  illustration: 


Alt  1 

Alt  2 

Alt  3 

Alt  4 

V0(x0) 

0.6 

0.9 

0.1 

0.4 

VI  (xl) 

0.8 

0.5 

0.6 

0.95 

V2(x2) 

0.9 

0.8 

0.7 

0.6 

V3(x3) 

0.9 

0.9 

0.3 

0.2 

Rank(K) 

2 

1 

4 

3 

Figure  3.11:  Data  Ex.  1 


S,  =Sw/ 

j= 0 

Sj  =  W10*W20  *.6  + W10  *W2l  *.8  +  Wl2  *W2 3  *  .9  +  W12  *W24  *.9 
S2=  Wl0*W20*.9  +  Wl0*W21  * .5  +  W12  *W2 3  *.8  +  Wn  *W24  *.9 
S3  =  W10*W20*.1  +  W10*W21  *.6  +  W12  *W23  *.7  +  Wl2*W24*.3 
S  4=  Wl0*W20  *  .4  +  W10  *W2l  *  .95  +  Wl2  *W23*.6  +  WI2  *W24*.  2 

Step  3:  Error  Function 

The  error  function  would  be  varied  to  see  how  it  effects  the  outcome,  but  an  example 
where  no  emphasis  is  placed  on  priority  of  matched  rankings  is  as  follows: 

MinE  = 

(|  Rankuk  -RankuL  \  +  \  Rank23  -Rank2lc  \  +  \  Rank33  -  Rank33  \  +  |  Rank43  -  Rank43  |) 

Step  4:  Formulate  Optimization 

Using  the  known  rankings,  the  entire  optimization  would  be: 

Min  Z  =  (|  2  -  Rankuc  |  +  1 1  -  Rank21c  \  +  \  4  -  Rank3lc  |  +  |  3  -  Rank 41‘  |) 
subject  to: 

^10+^11=1 


39 


r20+r2l=l 


^+^4=1 

WI0,W20,WU,W21,W23,W24<1 
W\0 ,  W20  ,Wn,  W21 ,  W23 ,  W24  >  0 

Step  5:  Solve  for  optimal  weights 

Excel’s  Evolutionary  Solver  was  the  optimization  package  chosen  for  this  analysis. 

Within  Excel,  the  weights  of  the  hierarchy  are  changed  for  a  specified  number  of 
iterations  or  candidate  solutions,  with  the  goal  of  minimizing  the  objective/  error 
function.  With  each  different  run  the  weights  vary  slightly  (because  of  the  starting 
conditions  and  inherent  randomness  of  the  genetic  algorithm),  however  the  general 
weighting  scheme  remains  the  same.  For  the  above  example,  one  of  the  optimal  solutions 
is  shown  below: 


Alt  1 

Alt  2 

Alt  3 

Alt  4 

V0(x0) 

0.6 

0.9 

0.1 

0.4 

VI  (xl) 

0.8 

0.5 

0.6 

0.95 

V2(x2) 

0.9 

0.8 

0.7 

0.6 

V3(x3) 

0.9 

0.9 

0.3 

0.2 

Rank(K) 

2 

1 

4 

3 

Score 

0.781327 

0.79849 

0.403613 

0.517833 

Rank(C) 

2 

1 

4 

3 

Error 

0 

Value  Weight 

Measure  Weight 

0 

0.512608 

0.651107 

1 

0.486741 

0.349608 

2 

0.599959 

3 

0.399202 

Figure  3.12:  Output  Ex.  1 
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Step  6:  Transpose  these  weights  into  the  hierarchy: 


This  output  translates  to  the  optimal  weighting  of  the  hierarchy  which  minimizes  the 
error  function: 


Figure  3.13:  Weighted  Hierarchy  Ex.  1 
Step  7:  Use  Hierarchy  To  Rank  Current  Period 

This  hierarchy,  with  completed  weights  may  be  used  to  score  a  current  list  of  alternatives, 
and  predict  the  ordinal  outcome  of  these  alternatives. 

3.3:  Example  2 

The  previous  example  was  trivial.  Now  consider  the  following  example,  using 
fictitious  data  once  again,  but  now  expanding  the  area  of  interest  over  four  time  periods, 
which  will  be  denoted  by  quarters  1-4. 

Step  1:  Hierarchy 
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Figure  3.14:  Hierarchy  Ex.  2 
Step  2:  Obtain  Historical  Data 
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Quarter  1 

Alt  A 

Alt  B 

Alt  C 

AltD 

VO(xO) 

0.6 

0.9 

0.1 

0.4 

VI  (xl) 

0.8 

0.5 

0.6 

0.95 

V2(x2) 

0.9 

0.8 

0.7 

0.6 

V3(x3) 

0.9 

0.9 

0.3 

0.2 

Rank(K) 

2 

1 

4 

3 

Quarter  2 

Alt  A 

Alt  B 

Alt  C 

AltD 

VO(xO) 

0.4 

0.4 

0.7 

0.2 

VI  (xl) 

0.7 

0.5 

0.3 

0.8 

V2(x2) 

0.5 

0.5 

0.5 

0.5 

V3(x3) 

0.4 

0.5 

0.7 

0.6 

Rank 

4 

3 

1 

2 

Quarter  3 

Alt  A 

Alt  B 

Alt  C 

AltD 

VO(xO) 

0.6 

0.5 

0.9 

1 

VI  (xl) 

0.5 

0.4 

0.8 

0.3 

V2(x2) 

0.8 

0.7 

0.6 

0.7 

V3(x3) 

0.6 

0.6 

0.8 

0.8 

Rank 

3 

4 

2 

1 

Quarter  4 

Alt  A 

Alt  B 

Alt  C 

AltD 

VO(xO) 

0.9 

0.6 

0.8 

0.8 

VI  (xl) 

0.7 

0.6 

0.5 

0.4 

V2(x2) 

0.2 

0.5 

0.6 

0.1 

V3(x3) 

1 

0.1 

0.5 

0.7 

Rank 

1 

4 

3 

2 

Figure  3.15:  Data  Ex.  2 

As  you  can  see  in  the  formulation  below,  scoring  must  be  done  for  each  quarter,  and  the 
error  function  must  be  expanded  to  include  all  quarters,  but  the  same  fundamentals  hold: 

Si'Ew/'r/i,) 

j= 0 

$a\ =  Wl0  *  W20  * .6  +  W10  *  W21  * .8  +  W12  *  W23  * .9  +  W12  *  W24  * .9 
SBl=  W10  * W20  * .9  +  W10  * W21  * .5  +  Wn  * W2i  * .8  +  W12  * W24  * .9 
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SC4  =  WX0*W2()*  .8  +  Wl0*W2X*  .5  +  Wn*W23  *.6  +  W12  *W24  ■ 

SD4=W10*W20*.8  +  W10*W21*.4  +  W12*W23*.1  +  W12*W24-- 

Step  3:  Error  Function 

(|  2  -  RankAlc  |  +  1 1  -  Rankmc  \  +  |  4  -  Rankcic  |  +  |  3  - 
(14  —  Rank ,2  1  +  13  -RankR2  I  +  1 1  -  Rankr2c  1  +  12 
(|  3  -  RankA3  |  +  |  4  -  RankB3  |  +  |  2  -  RankC3  |  +  1 1 
(|  1  -  RankA4  \  +  |  4  -  RankB4  |  +  |  3  -  RankC4c  \  +  |  2 

Step  4:  Set  up  optimization 

(|  2  -  Rank A 3  |  +  1 1  -  RankmL  \  +  |  4  -  Rankclc  |  +  |  3  - 
(14  —  Rank ,/  1  +  13  -RankR2  I  +  1 1  -  RankC2  1  +  12 
(|  3  -  RankA3  \  +  |  4  -  RankB3  |  +  |  2  —  RankC3  \  +  1 1 
(|  1  -  RankA4  |  +  |  4  —  RankB4  |  +  |  3  -  RankC4  \  +  |  2 

subject  to: 

^10+^11=1 

r2o+r2,  =1 

^+^4=1 

W10,W20,Wn,W2l,W23,W24<l 

Ww,W2O,Wn,W2l,W23,W24>0 


Step  5:  Solve  for  optimal  weights 


15 .5 

K1 

-  Rankmc  |)  + 

-  Rankm  |)  + 

-  Rankmc  |)  + 

-  RankD4  |) 

-  Rankmc  |)  + 

-  RankD2  |)  + 

-  Rankmc  |)  + 

-  RankD4  |) 
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After  running  the  Premium  Solver  on  this  problem,  one  of  the  optimal  solutions  was  as 
follows: 


Quarter  1 
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Error 

0 
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0 
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Rank(S) 

3 

4 

2 

1 

Error 

0 
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Category  Weight 

Measure  Weight 

1 

0.383787 

0.788928 

2 

0.61437 

0.212117 

3 

0.263592 

4 

0.735885 

Figure  3.16:  Output  Ex.  2 
Step  6:  Transpose  Weights  Into  Hierarchy 

This  output  translates  to  the  optimal  weighting  of  the  hierarchy  which  minimizes  the 
error  function: 


Figure  3.17:  Complete  Hierarchy  Ex.  2 

This  weighting  minimized  the  difference  between  known  and  calculated  rankings  over 
each  of  these  four  time  periods.  In  this  particular  example,  the  weighting  was  effective  to 
the  extent  that  scoring  any  of  the  four  quarters  with  the  above  weighting  scheme,  resulted 
in  an  error  of  zero. 

Step  7:  Use  Hierarchy  To  Rank  Current  Period 

This  hierarchy,  with  completed  weights  may  be  used  to  score  a  current  list  of 
alternatives,  and  predict  the  ordinal  outcome  of  these  alternatives. 
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3.4  Summary 

This  chapter  has  provided  detailed  methodology  behind  using  optimization  to 
select  the  weights  of  a  VFT  hierarchy.  The  mathematical,  expandable  example  should 
prove  useful  as  a  shell  for  any  problem  using  this  7-step  methodology.  Chapter  4  will 
implement  this  methodology  on  solving  the  problem  of  deciding  the  relative  importance 
of  military  construction  (MILCON)  projects. 
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Chapter  4.  Results  and  Analysis 


4.0  Overview 

This  chapter  will  provide  an  example  of  using  optimization  to  detennine  the 
weighting  scheme  of  a  value  focused  thinking  hierarchy.  This  example  is  a  broad 
overview  of  the  process,  and  although  it  highlights  the  civil  engineering  arena,  the  goal  of 
this  chapter  is  to  clearly  depict  how  this  methodology  could  apply  to  most  any  area  of 
study. 

4.1  MILCON  Project  Selection  as  an  Example 

“The  mission  of  HQ  USAF/ILE,  the  Office  of  The  Civil  Engineer,  is  to  provide 
the  bases,  infrastructure  and  facilities  necessary  to  support  the  global  engagement  of  air 
and  space  forces  across  the  spectrum  of  conflict”  (Installation).  AIR  FORCE 
PAMPHLET  32-1004,  VOLUME  1,  breaks  down  Air  Force  Civil  Engineering  in  the 
following  manner: 
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Figure  4.1:  Base  Civil  Engineering  Hierarchy 

Within  this  hierarchy,  the  Engineering  Flight  is  the  facet  in  charge  of  any  major 
construction  on  an  Air  Force  installation.  Each  base  is  allocated  a  certain  budget  to  take 
care  of  routine,  small  construction  and  maintenance.  When  a  major  project  must  be 
completed,  however,  approval  for  funding  must  go  through  the  “MIFCON”  (Military 
Construction)  approval  process.  A  military  construction  project,  as  defined  by  Air  Force 
Instruction  32-1021  is: 

Military  construction  as  defined  in  the  law  includes  any  construction, 
development,  conversion,  or  extension  of  any  kind  carried  out  with  respect 
to  a  military  installation.  It  includes  all  construction  work  necessary  to 
produce  a  complete  and  usable  facility  or  a  complete  and  usable 
improvement  to  an  existing  facility.  Authority  to  carry  out  a  military 
construction  project  includes  authority  for  surveys  and  site  preparation; 
acquisition,  conversion,  rehabilitation,  or  installation  of  facilities; 
acquisition  and  installation  of  equipment  and  appurtenances  integral  to  the 
project;  acquisition  and  installation  of  supporting  facilities  (including 
utilities)  and  appurtenances  incident  to  the  project;  and  planning, 
supervision,  administration,  and  overhead  incident  to  the  project.  (16) 
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The  current  monetary  threshold  for  a  MILCON  project  is  $750,000  (Dempsey).  The  time 
it  takes  from  initial  proposal  to  approval  and  the  start  of  construction  is  typically  a 
minimum  of  two  years  (Dempsey).  The  MILCON  process  flow-chart  as  given  by  The 
United  States  Air  Force  Program  Manager’s  Guide  for  Design  and  Construction,  is 
shown  in  figure  4.2: 
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The  first  steps  of  the  MILCON  process  are  summarized  by  the  following  descriptions 
taken  from  AFI  32-1021: 


Project  Identification  and  Prioritization:  Installations  must  identify 
facility  needs,  and  detennine  which  cannot  be  met  with  existing  facilities. 
Installation  commanders  assisted  by  the  base  Facilities  Board  will  review, 
validate  and  prioritize  the  installation  MILCON  facility  requirements.  In 
accordance  with  schedules  established  by  the  MAJCOM,  the  Civil 
Engineering  Squadron  Commander  will  prepare  and  submit  the  DD  Fonns 

1391,  1391C,  FY _ Military  Construction  Project  Data,  AF  Form 

1 178,  Project  Cost  Estimate  Summary,  and  enter  the  project  into  the 
Program,  Design  and  Construction  data  system.  A  Parametric  Cost 
Estimate  and  economic  analyses  (AFI  65-501,  Economic  Analysis  and 
Program  Evaluation  for  Resource  Management )  must  also  be  submitted  to 
the  host  or  tenant  MAJCOM  as  appropriate.  Based  on  the  Installations’ 

submittals  HQ  USAF/CECD  will  complete  the  DD  Form  1390,  FY  19 _ 

Military  Construction  Program  (S&U,  HQ  USAF/CECD,  1260  Air 
Force  Pentagon,  Washington,  DC  20330-1260,  DoD  Financial 
Management  Regulation,  Volume  2B,  Chapter  6.  )  For  ANG  projects,  the 
Civil  Engineering  Squadron  Commander  submits  projects  to  NGB/CEDD. 

DD  Forms  1390S1  and  1390S2,  FY _ Guard  and  Reserve  Military 

Construction,  will  be  submitted  to  Headquarters  Air  Force  Reserves  for 
their  projects. 

MAJCOM  MILCON  Budget:  The  MAJCOM  civil  engineer  and 
functional  staff  will  review  and  verify  data  submitted  by  the  command’s 
installations.  The  MAJCOM  commander  will  evaluate  and  prioritize  the 
projects  and  develop  and  submit  a  command  program  (via  hard  copy  and 
Programming,  Design,  and  Construction  computer  system)  to  HQ 
USAF/CEC  that  matches  the  MILCON  program  element  totals  established 
by  HQ  USAF  Resource  Allocation  Teams.  MAJCOMs  may  submit 
projects  over  and  above  these  MILCON  totals  to  HQ  USAF/CEC  who  will 
review  and  validate  them.  Additional  funds,  however,  must  be  requested 
by  MAJCOM/XPs  through  the  HQ  USAF  Resource  Allocation  Teams. 

Air  Force  MILCON  Budget:  HQ  USAF/CEC,  in  conjunction  with  other 
HQ  USAF  functional  offices  will  review  each  project  in  detail  and 
validate  need,  engineering  feasibility,  economic  benefits,  compliance  with 
Air  Force  objectives,  and  project  cost.  HQ  USAF/CEC  will  authorize 
initiation  of  design  on  validated  projects.  Notification  per  10  U.S.C.  2807 
may  be  required  prior  to  award  of  an  architect-engineer  design  contract. 
(18) 
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When  the  Air  Staff  reviews  the  MILCON  project  requests,  the  projects  go  through 
a  prioritization  process.  The  goal  of  this  process  is  to  rack  and  stack  all  of  the  projects  in 
order  of  their  importance/priority.  With  this  rank-ordered  list,  funds  may  be  allocated  in 
a  manner  the  best  suits  the  Air  Force’s  needs.  This  prioritization  process  for  MILCON 
projects  has  historically  been  done  so  in  the  following  manner  (Rafferty  2002): 


Figure  4.3:  MILCON  Integrated  Priority  Listing 

The  problem  with  this  priority  listing  is  that  the  corporate  adjustments  have  dominated 
the  prioritization  process.  The  initial  purpose  of  the  corporate  adjustments  was  to 
override  the  prioritization  process  for  a  MILCON  project  that  was  absolutely  critical. 
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Over  time,  unfortunately,  the  number  of  projects  that  are  “corporately  adjusted”  has  risen 
to  over  90%.  With  these  statistics,  if  one  wishes  to  have  any  chance  of  their  MILCON 
project  being  approved,  the  solicitation  for  the  project  must  be  done  so  in  such  a  manner 
deserving  of  corporate  adjustment.  Otherwise,  it  will  be  one  of  the  only  projects  that  are 
actually  prioritized  through  the  model,  and  the  odds  of  receiving  funding  is  dismal. 

Major  Andre  Dempsey  is  proposing  a  new  way  to  prioritize  MILCON  projects.  Using 
Value  Focused  Thinking,  all  projects  will  be  evaluated  by  a  hierarchy  with  the 
fundamental  objective  of  determining  “what  is  valued  in  a  MILCON  project.”  The 
resulting  scores  of  the  alternatives  will  detennine  the  priority  of  the  MILCON  projects. 
Using  Dempsey’s  hierarchy,  and  the  abundance  of  historical  data  available  for  the 
measures  of  his  hierarchy,  the  following  sections  will  illustrate  this  real-world  example  of 
using  the  7-step  process  to  solve  for  the  optimal  weights  of  a  VFT  hierarchy. 

4.2  Example  1:  Solving  For  Historical  Weights 

In  this  example,  the  “known”  rankings  are  the  rankings  previously  determined 
during  the  prioritization  process  (the  rankings  the  board  made).  Therefore,  this  analysis 
will  show  which  areas  the  selection  board  valued  highly,  and  which  areas  they  did  not. 
This  type  of  implementation  is  useful  in  illustrating  parts  of  the  decision  that  were  left 
out,  but  in  reality  should  be  influential.  These  weights  will  be  compared  to  those 
determined  by  Dempsey,  that  he  feels  best  allocate  available  weight  among  the  values. 
This  will  highlight  the  major  value  gaps  with  the  old  prioritization  process. 

4.2.1  Step  1:  Develop  the  Hierarchy 
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Force  structure 


Fundamental  Objective:  What  is  Valued  in  a  MILCON  Project? 

Maj.  Andre  Dempsey  developed  the  following  hierarchy,  broken  down  into  its  three 
branches: 


Operational 


i 


X 


Efficiencies 


Select  Right  MILCON 
Projects 


Operational 

Support 


Resources 


Economics 


Figure  4.4:  MILCON  Hierarchy  (Branch  1) 


Quality  of  Life 
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Figure  4.5:  MILCON  Hierarchy  (Branch  2) 


Figure  4.6:  MILCON  Hierarchy  (Branch  3) 


56 


4.2.2  Step  2:  Obtain  Historical  Data  for  Measures 


Since  the  historical  data  for  this  example  covers  200  alternates,  the  large  list  of 
data  may  be  found  in  Appendix  A.  This  data  includes  the  y-value  scores  after  translation 
by  the  SDVFs,  and  the  alternates  are  in  rank  ordered  score  from  one  through  two 
hundred. 

4.2.3  Step  3:  Develop  Error  Function 

With  only  one  time  period,  and  no  priority  on  matching  up  any  rankings  of 
alternates  over  others,  the  error  function  is: 

Min  E  =  (|  Rankuk  -Ranknc  |  +  |  Rank2k  -  Rank2lc  |  +  .  .  .+  |  Rank200k  -  Rank200c  |) 

4.2.4  Step  4:  Create  Optimization 

period  1  Alt  200 

Minimize  E  =  Z  Z  |  Rank  A  -  Rank  A  \ 

j = period  1  i=Alt\ 

Subject  to:  =  1  V  JV(i+1)  v  e  Dtj  (the  sum  of  all  local  weights  =  1,  i  =  1,2, 3, 4, 5) 

0  <  Wj  /  <1  (all  weights  must  be  between  zero  and  one) 

Definitions  for  any  of  the  above  calculations  may  be  reviewed  in  chapter  3. 


4.2.5  Step  5:  Solve  for  Optimal  Weights 

This  step  of  the  VFT  process  was  done  in  Excel’s  Premium  Solver.  Solver  was 
unable  to  determine  a  weighting  scheme  that  resulted  in  an  absolute  error  =  0  for  all  two 
hundred  alternatives.  This  task  may  or  may  not  be  possible,  but  from  the  areas  of  the 
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solution  space  explored  by  Solver,  the  local  minimums  found  for  50  independent  runs 
may  be  viewed  in  Appendix  B. 

4.2.6  Step  6:  Integrate  Weights  Into  Hierarchy 

Table  4.5  provides  the  weighting  for  the  complete  hierarchy,  including  the  local 
weights  of  the  values,  and  global  weights  of  the  measures.  These  weights  were  chosen 
for  their  low  absolute  error  in  difference  between  “known”  and  “calculated”  rankings  of 
alternatives: 


0.0279 

0.9356 

0.0319 

0.5029 

0.4979 

0.2558 

0.2528 

0.2319 

0.2545 

0.5071 

0.4921 

0.4049 

0.5959 

0.2767 

0.7241 
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0.0567 

0.9399 

1.0000 

0.0884 

0.9114 

0.0886 

0.0693 

0.9469 

0.0532 

1.0000 

0.4494 
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1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

0.0226 

0.8908 

0.0884 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

0.0057 

0.0038 

0.0070 

0.0038 

0.0101 

0.2342 

0.0049 

0.0134 

0.0050 

0.1981 

0.0197 

0.2170 

0.0210 

0.2170 

0.0014 

0.0011 

0.0149 

0.0008 

Figure  4.7:  Weights  of  Hierarchy  (Example  1) 


4.2.7  Step  7:  Use  Hierarchy  to  Rank  Alternatives 

The  complete  hierarchy  may  now  be  used  to  rank  a  current  list  of  alternatives. 

This  list  would  be  the  best  guess  at  how  the  MICON  projects  would  be  ranked  by  the 
panel,  if  they  continue  their  prioritization  process  under  the  same  trends. 

4.2.8  Summary  of  Findings  From  Example  1 

Apparent  from  the  derived  weights  for  the  chosen  optimization  run,  historically 
the  MILCON  prioritization  process  has  been  driven  by  Deficit  IRR,  Mission  Panel 
Points,  and  Correcting  an  AFTP  Deficiency.  Corporate  adjustment  was  mentioned  earlier 
as  being  a  necessity  in  the  prioritization  process,  if  a  project  had  hope  of  approval.  Data 
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was  only  available  on  the  first  200  alternatives,  all  which  were  corporately  adjusted, 
therefore  it  did  not  return  a  high  weight  in  the  optimization  process.  Once  again,  weights 
differ  between  runs  due  to  the  convergence  on  multiple  local  minimums.  The  distribution 
of  the  weights  over  fifty  runs  proved  to  be  normal.  This  is  desirable,  for  it  shows  that  the 
weights  from  the  different  runs  follow  a  pattern,  with  most  runs  falling  very  close  to  the 
mean.  Histograms  showing  the  distributions  around  the  first  tier  of  values  are: 


k-"1 

_ 

Distribution:  Normal 
Expression:  NORM(0. 175,  0.0662) 
Square  Error:  0.032339 


Figure  4.8:  Distribution  of  Weights  (Efficiencies  Value) 


Distribution:  Normal 
Expression:  NORM(0.59, 0.0904) 

Square  Error:  0.014304 

Figure  4.9:  Distribution  of  Weights  (Operational  Support  Value) 
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Distribution:  Normal 
Expression:  NORM(0.235, 0.0446) 

Square  Error:  0.009226 

Figure  4.10:  Distribution  of  Weights  (Quality  of  Life  Value) 

4.3  Example  2:  Solving  For  Known  Weights 

In  this  example,  using  the  hierarchy  with  weights  determined  by  Dempsey,  the 
alternatives  will  be  re-scored,  and  the  7-step  process  will  be  used  to  determine  how  close 
the  optimization’s  weights  come  to  those  “known”  weights  that  Dempsey  used  to 
ordinally  rank  the  alternates. 

4.3.1  Step  1:  Develop  the  Hierarchy 

Fundamental  Objective:  What  is  Valued  in  a  MILCON  Project? 

The  following  hierarchy  was  developed  by  Maj.  Andre  Dempsey,  and  is  the  same 
hierarchy  as  in  Example  1 : 
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Improves  \  Force  structure 

IRR  7  l&fl  I  _ 


Figure  4.11:  MILCON  Hierarchy  (Branch  1) 


E 


Figure  4.12:  MILCON  Hierarchy  (Branch  2) 
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Figure  4.13:  MILCON  Hierarchy  (Branch  3) 


4.3.2  Step  2:  Obtain  Historical  Data  for  Measures 

The  list  of  historical  data  may  be  found  in  Appendix  D.  This  data  includes  the  y- 
value  scores  after  translation  by  the  SDVFs,  and  the  alternates  are  in  rank  ordered  score 
(ranked  by  Dempsey’s  weighted  hierarchy),  for  each  of  the  200  alternates. 

4.3.3  Step  3:  Develop  Error  Function 

With  only  one  time  period,  and  no  priority  on  matching  up  any  rankings  of 
alternates  over  others,  the  error  function  is: 

MinE  =  (|  Rankuk  -Rankn°  |  +  |  Rank2k  -Rank2Xc  |  +.  .  .+  |  Rank200k  -Rank200c  |) 

4.3.4  Step  4:  Create  Optimization 

period  1  Alt  200 

Minimize  E  =  Z  Z  |  Rank —Rank A  \ 

j = period  1  i=Alt\ 
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Subject  to:  =  1  V  W(i+1)  v  e  ZU  (the  sum  of  all  local  weights  =  1,  i  =  1,2,3, 4, 5) 


0<Wij  <1  (all  weights  must  be  between  zero  and  one) 

Definitions  for  any  of  the  above  calculations  may  be  reviewed  in  chapter  3. 


4.3.5  Step  5:  Solve  for  Optimal  Weights 

This  step  of  the  VFT  process  was  done  in  Excel’s  Premium  Solver.  The  response 
surface  produced  by  the  19  measures,  27  values,  and  200  alternates  was  rather  large. 
Unlike  the  earlier  example  problems  in  Chapter  3,  where  Solver  was  able  to  quickly 
converge  to  the  globally  optimal  solution,  Solver  found  good  results  with  each  run, 
however  the  globally  optimal  solution  of  Error  =  0  was  never  reached  (Error  =  0  if 
Dempsey’s  weights  were  inputted).  Each  run  converged  on  a  locally  optimal  solution, 
providing  good  weights  for  the  hierarchy,  but  because  the  focus  was  too  quickly 
narrowed  into  this  local  area  of  interest,  Error  =  0  was  never  found.  A  different  genetic 
algorithm  with  more  variable  parameters  such  at  crossover  rate,  would  likely  find  the 
optimal  solution,  but  the  complexity  of  this  response  surface  proved  too  difficult  for 
Premium  Solver’s  Evolutionary  Solver.  The  solved  weights  for  50  independent  runs  may 
be  found  in  Appendix  E. 
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4.3.6  Step  6:  Integrate  Weights  Into  Hierarchy 


Both  the  “known”  and  “calculated”  weights  are  show  below,  for  the  sake  of 
comparison.  This  run  of  “calculated”  weights  as  chosen  for  their  lowest  absolute  error  in 
difference  between  “known”  and  “calculated”  rankings  of  alternatives: 
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Figure  4.14:  Known  and  Calculated  Weights  (Example  2) 


4.3.7  Summary  of  Findings  From  Example  2 

Although  Solver  never  converged  on  Error  =  0,  the  derived  weights  for  the 
hierarchy  came  very  close  to  the  true  weights  that  Dempsey  used  to  score  the  alternates. 
At  a  minimum,  they  would  provide  an  analyst  such  as  Dempsey  with  an  excellent  “first 
cut”  at  the  process  of  weighting  the  hierarchy.  As  can  be  seen  in  Figure  4.9,  the  derived 
weights  at  each  tier  showed  progression  in  accuracy  with  each  tier.  It  is  also  important  to 
see  that  the  relative  weights  at  each  tier  were  proportionally  correct  (i.e.  tier  1-  Value  2  > 
Value  3  >  Value  1).  At  the  tier  of  measures,  the  total  absolute  error  between  known  and 
derived  weights  is  only  .153.  This  shows  valuable  insight  into  the  process. 
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As  in  Example  1 ,  the  distribution  of  the  weights  over  fifty  runs  proved  to  be 
normal,  and  the  weights  from  the  different  runs  fall  very  close  to  the  mean.  Histograms 
showing  the  distributions  around  the  first  tier  of  values  are: 


Distribution:  Normal 
Expression:  NORM(0.256, 0.0341) 
Square  Error:  0.029198 


Figure  4.15:  Distribution  of  Weights  (Efficiencies  Value) 


Distribution:  Normal 
Expression:  NORM(0.443, 0.0384) 

Square  Error:  0.040188 

Figure  4.16:  Distribution  of  Weights  (Operational  Support  Value) 
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Distribution:  Normal 
Expression:  NORM(0.3, 0.018) 

Square  Error:  0.000595 

Figure  4.17:  Distribution  of  Weights  (Quality  of  Live  Value) 

4.4  Summary 

This  chapter  has  shown  a  large-scale,  real  world  implementation  of  using 
optimization  to  select  the  weights  for  a  VFT  hierarchy.  The  results  were  positive  taking 
into  consideration  the  huge  response  surface  created,  and  the  inability  to  customize  the 
GA  portion  of  the  Solver  to  meet  the  needs  of  this  solution  space.  This  implementation 
has  given  an  idea  of  the  diverse  purposes  this  methodology  could  be  used  for,  and  the 
insight  the  results  can  provide. 
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Chapter  5.  Discussion 


5.0  Introduction 

This  chapter  briefly  summarizes  the  proposed  optimization  based  methodology 
for  detennining  the  weights  of  a  value  focused  thinking  hierarchy.  The  limitations  of  this 
methodology  will  be  discussed,  and  recommendations  will  be  made  for  future  research  in 
this  area. 

5.1  Conclusions 

Value  Focused  Thinking  is  a  strong,  growing  subset  of  Decision  Analysis.  This 
strategic  way  of  making  difficult  decisions  has  set  the  standard  in  multiple  industries. 

One  of  the  major  drawbacks  of  VFT,  however,  is  the  great  amount  of  time  and  effort  it 
takes  by  decision  makers  and/or  subject  matter  experts  to  provide  insight  into  the  process 
being  studied.  This  is  especially  true  when  detennining  the  appropriate  weights  for  a 
hierarchy.  Conventional  weighting  techniques  often  require  unrealistic  time  demands  on 
these  DMs/SMEs,  especially  in  circumstances  necessitating  high  ranking  military 
officers,  heads  of  businesses,  etc. . .  When  historical  data  permits,  this  step  of  the  VFT 
process  can  be  greatly  reduced,  or  even  omitted,  by  solving  for  the  historically  optimal 
weights,  that  minimize  an  error  function  based  on  the  difference  between  known  and 
predicted  perfonnance.  Depending  on  the  complexity  of  the  process,  and  size  of  the 
solution  space,  the  process  can  provide  optimal  weights  for  the  hierarchy,  or  at  a 
minimum,  a  far  superior  starting  point  for  the  discussion  on  weighting,  hence  reducing 
the  time  demands  on  the  DMs/SMEs. 


67 


5.1  Limitations 


The  methodology  explored  in  this  thesis  proved  to  be  very  successful.  Great 
insight  was  gained  into  the  MILCON  process  by  using  available  historical  information. 
This  use  of  past  knowledge,  combined  with  modem  optimization  techniques,  was  able  to 
provide  information  into  how  MILCON  projects  were  prioritized  by  a  panel  (by  knowing 
the  relative  tradeoffs  in  values  of  the  hierarchy).  When  this  methodology  was  tested  on 
the  MILCON  example  with  “known”  weights,  beneficial  information  into  these  tradeoffs 
was  also  gained,  despite  never  converging  on  the  perfect  weights. 

Frontline’s  Genetic/Evolutionary  Algorithm  Solver  was  the  optimization  package 
used  due  to  the  benefits  it  showed  for  this  particular  methodology.  By  combining  a 
genetic/evolutionary  algorithm  with  standard  linear  and  non-linear  program,  the  solution 
space  was  best  explored.  The  chaotic  nature  of  many  real  world  problems,  such  as  the 
MILCON  example  in  Chapter  4,  require  a  tool  such  as  a  GA  to  provide  quality  results. 
This  tool  easily  converged  on  “known”  optimal  weights  of  many  dummy  problems,  both 
small  and  large.  Unfortunately  in  the  case  of  the  MILCON  example,  the  response  surface 
proved  to  be  too  large  and  chaotic  for  Premium  Solver.  Optimality  was  never  reached, 
not  due  to  a  problem  with  the  methodology,  but  due  to  the  limitations  of  the  optimization 
package.  The  “tweaking”  of  the  GA  in  Solver  is  tightly  controlled,  not  allowing  enough 
exploration  of  the  solution  space  before  converging  on  a  local  minimum.  This  in  turn 
limited  the  success  of  the  optimization,  and  furthermore  detracted  from  the  success  of  this 
methodology. 
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5.2  Future  Research 


Ones  success  using  this  methodology  on  very  large  problems  would  greatly 
increase  by  using  a  more  user  defined  genetic  algorithm.  The  freedoms  provided  by  this 
will  allow  you  to  tailor  the  GA  to  the  needs  of  your  response  surface.  Future  research 
could  be  focused  in  this  area,  as  well  as  closely  looking  at  the  effects  size  of  the  hierarchy 
and  number  of  time  periods  have  on  the  success  of  the  process. 

5.4  Summary 

The  proposed  methodology  provides  a  beneficial  weighting  scheme  for  a  VFT 
hierarchy  when  input  from  DMs/SMEs  is  limited  or  unavailable.  This  application 
provides  the  VFT  community  with  a  new  way  to  weight  a  hierarchy,  when  conventional 
methods  are  not  possible.  Using  this  methodology,  instead  of  making  the  mistake  that 
George  Santayana  warns  against,  “Those  who  ignore  history  are  doomed  to  repeat  it”,  we 
can  use  our  knowledge  of  the  past  to  provide  insight  into  the  future,  and  hope  the  history 
indeed  repeats  itself. 
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Improves  Force  structure 

IRR  7  _ 


Appendix  A 


This  appendix  shows  the  individual  tiers  of  the  MILCON  hierarchy. 


First  Tier  of  Hierarchy 


Second  Tier  of  Hierarchy 


E 
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Third  Tier  of  Hierarchy 
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Base  Population 


Appendix  B 


This  appendix  contains  the  data  for  the  measures  of  the  MILCON  hierarchy: 
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Appendix  C 


This  appendix  contains  the  output  from  50  runs  of  the  model: 


Average 

1 

2 

3 

4 

5 

6 

Face  Structure 

0.0299628 

0.013771 

0.012668 

0.066181 

0.046262 

0.023875 

0.011032 

Consol  idatbn 

0.0209869 

0.008476 

0.035995 

0.013682 

0.031062 

0.014375 

0.011953 

Fcorprint  Reduction 

0.023668 

0.006448 

0.029903 

0.096293 

0.048155 

0.016201 

0.018491 

Joint  Use 

0.0258702 

0.018455 

0.037726 

0.066854 

0.029732 

0.013522 

0.006973 

Payback 

0.0574659 

0.033835 

0.075233 

0.099812 

0.073895 

0.053317 

0.049886 

Deficit  IRR 

0.1459364 

0.165428 

0.129895 

0.031241 

0.060004 

0.17104 

0.191052 

Rest.  &  Modern.  IRR 

0.0180274 

0.007361 

0.028327 

0.052092 

0.08649 

0.01082 

0.008198 

Design  Build 

0.0282878 

0.030229 

0.030829 

0.018915 

0.050161 

0.023478 

0.032524 

Years  to  lOC/Need  date 

0.0119944 

0.009517 

0.025437 

0.020797 

0.031564 

0.003525 

0.019844 

Mission  Pannel  Points 

0.0410921 

0.051544 

0.0138 

0 

0.021059 

0.035642 

0.033016 

Installation  CC's  Prbrity 

0.0798081 

0.079206 

0.086934 

0.043621 

0.043709 

0.090385 

0.113823 

Corrects  ATFP  Deficiency 

0.1031996 

0.169137 

0.073451 

0.083333 

0.107422 

0.146681 

0.004105 

Direct  Operational  Support 

0.0715769 

0.087958 

0.060672 

0.067334 

0.0636 

0.066631 

0.085958 

Warfighting  Enablers 

0.0897665 

0.08448 

0.096481 

0.015999 

0.082893 

0.084823 

0.113098 

Facility  Age 

0.0560299 

0.05432 

0.044055 

0.134668 

0.052039 

0.051106 

0.064406 

Eliminates  Safety  Violation 

0.0378305 

0.022137 

0.042893 

0.098678 

0.074675 

0.014982 

0.012783 

Support  Facility 

0.0791751 

0.0825 

0.059018 

0.096662 

0.070424 

0.075961 

0.093507 

Base  Populatbn 

0.0333885 

0.016527 

0.05509 

0.070005 

0.049441 

0.040847 

0.042238 

Error 

7710.32 

7118 

8172 

9582 

9360 

7324 

7432 

Efficiencies 

0.107518 

0.225925 

0.333333 

0.213367 

0.133527 

0.1138 

op  suppt 

0.684252 

0.545892 

0.333333 

0.546903 

0.632738 

0.601068 

QOL 

0.205521 

0.228203 

0.333333 

0.23973 

0.233574 

0.285132 

operational 

0.513717 

0.500018 

0.5 

0.514324 

0.500055 

0.5 

resources 

0.48631 

0.500013 

0.5 

0.485676 

0.499964 

0.5 

readiness 

0.252496 

0.28985 

0.25 

0.26786 

0.287406 

0.331837 

responsiveness 

0.248647 

0.28763 

0.25 

0.26786 

0.240186 

0.33081 

secu  rity 

0.247185 

0.134552 

0.25 

0.196419 

0.23182 

0.00683 

missions 

0.251798 

0.287872 

0.25 

0.26786 

0.240214 

0.33081 

work  QOL 

0.517198 

0.500003 

0.5 

0.5 

0.500246 

0.524153 

sense  of  community 

0.481837 

0.500011 

0.5 

0.5 

0.499968 

0.475847 

right  place 

0.24933 

0.112142 

0.397085 

0.421558 

0.357567 

0.193891 

right  size 

0.750879 

0.887827 

0.602915 

0.578442 

0.642226 

0.806109 

joint  use 

0.352958 

0.333964 

0.401127 

0.286911 

0.202548 

0.122548 

econ 

0.647094 

0.665981 

0.598873 

0.713089 

0.798657 

0.876722 

reduce  deficit 

0.957499 

0.820941 

0.374892 

0.4096 

0.94054 

0.957862 

restore  and  mondernize 

0.042604 

0.179026 

0.625108 

0.5904 

0.059501 

0.0411 

ability  to  execute 

0.177672 

0.196343 

0.226985 

0.342413 

0.154486 

0.163568 

mission  timing 

0.822408 

0.803664 

0.773015 

0.657587 

0.851051 

0.836605 

combat  capability 

0.510514 

0.386083 

0.808009 

0.434151 

0.438381 

0.432297 

mission  support 

0.490324 

0.613954 

0.191991 

0.565849 

0.558071 

0.568789 

modern  facilities 

0.511027 

0.386103 

0.808009 

0.434151 

0.437388 

0.430945 

safe  facil 

0.208262 

0.375917 

0.592069 

0.622995 

0.128224 

0.085531 

support  facilities 

0.833098 

0.517225 

0.579973 

0.587526 

0.650462 

0.689178 

promotes  community 

0.166895 

0.482805 

0.420027 

0.412474 

0.349776 

0.311306 

Consol  idatbn 

0.204378 

0.358897 

0.136158 

0.489336 

0.33523 

0.260601 

Foorprint  Reduction 

0.155482 

0.2981  56 

0.958276 

0.758602 

0.377816 

0.403146 

Years  to  lOC/Need  date 

0.068019 

0.201585 

0.322838 

0.327657 

0.027254 

0.119289 

Installatbn  CC's  Prbrity 

0.566071 

0.688933 

0.677162 

0.453734 

0.698827 

0.684241 

Mission  Panel  Points 

0.368376 

0.109363 

0 

0.218609 

0.275571 

0.198472 
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7 

8 

9 

10 

11 

12 

13 

14 

15 

0.03909 

0.036583 

0.064541 

0.004855 

0.010208 

0.029324 

0.014956 

0.012741 

0.037597 

0.06556 

0.043255 

0.009378 

0.010816 

0.023096 

0.018555 

0.011196 

0.001237 

0.014011 

0.056681 

0.00923 

0.028119 

0.010954 

0.026821 

0.013791 

0.015552 

0.011296 

0.025083 

0.039667 

0.05847 

0.027817 

0.009031 

0.032244 

0.032137 

0.019276 

0.005009 

0.030808 

0.093717 

0.072911 

0.040293 

0.015924 

0.055931 

0.061989 

0.031989 

0.026362 

0.066863 

0.070761 

0.086735 

0.10768 

0.20486 

0.149012 

0.167996 

0.1738 

0.191983 

0.146495 

0.045855 

0.031884 

0.042675 

0.000813 

0.001279 

0.009154 

0.011264 

0.005784 

0.010456 

0.053793 

0.040649 

0.053727 

0.028972 

0.019106 

0.026972 

0.029915 

0.03844 

0.025722 

0.022196 

0.015958 

0.002575 

0.053254 

0.008781 

0.00835 

0.004765 

0.026955 

0.008284 

0.000616 

0 

0.051508 

0.052781 

0.055883 

0.044562 

0.06969 

0.044782 

0.015912 

0.040012 

0.062011 

0.042545 

0.071723 

0.067066 

0.095916 

0.077014 

0.08662 

0.10715 

0.116616 

0.118619 

0.150355 

0.1148 

0.137608 

0.023351 

0.126277 

0.113342 

0.060142 

0.054867 

0.061879 

0.063745 

0.078005 

0.068732 

0.084088 

0.06788 

0.092553 

0.072354 

0.061749 

0.05674 

0.08661 

0.127494 

0.081433 

0.092594 

0.114376 

0.104551 

0.08468 

0.067811 

0.068537 

0.04701 

0.040244 

0.051334 

0.062457 

0.042412 

0.054399 

0.064967 

0.09573 

0.063724 

0.031934 

0.012919 

0.027116 

0.021014 

0.016423 

0.012822 

0.01601 

0.080407 

0.086676 

0.096481 

0.079003 

0.071241 

0.0962 

0.07549 

0.090512 

0.076906 

0.042233 

0.044706 

0.014403 

0.025014 

0.029203 

0.027231 

0.032056 

0.025247 

0.052247 

9174 

8770 

8486 

7622 

7396 

7262 

7306 

7338 

7618 

0.266768 

0.262762 

0.176813 

0.056695 

0.197957 

0.1894 

0.103207 

0.062747 

0.198694 

0.466465 

0.474476 

0.60142 

0.731776 

0.588804 

0.553243 

0.674932 

0.70487 

0.531163 

0.266768 

0.262762 

0.221768 

0.210289 

0.212742 

0.254842 

0.221444 

0.232261 

0.2701  99 

0.5 

0.5 

0.61479 

0.558527 

0.55464 

0.500533 

0.502989 

0.499998 

0.508486 

0.5 

0.5 

0.38521 

0.440252 

0.445232 

0.496954 

0.496695 

0.499996 

0.491515 

0.25 

0.25 

0.25 

0.281098 

0.25529 

0.320188 

0.274109 

0.28057 

0.295486 

0.25 

0.25 

0.25 

0.281033 

0.255116 

0.318449 

0.268738 

0.279632 

0.295482 

0.25 

0.25 

0.25 

0.156879 

0.233708 

0.042208 

0.187096 

0.160799 

0.113226 

0.25 

0.25 

0.25 

0.281032 

0.255234 

0.319378 

0.270058 

0.279659 

0.295482 

0.540273 

0.5 

0.5 

0.504994 

0.527244 

0.515907 

0.514293 

0.500834 

0.521943 

0.459727 

0.5 

0.5 

0.494617 

0.472669 

0.484094 

0.485745 

0.498362 

0.478006 

0.293067 

0.278447 

0.593738 

0.153311 

0.092969 

0.309321 

0.2881 

0.406093 

0.372131 

0.706933 

0.721553 

0.406262 

0.850034 

0.907063 

0.690429 

0.712174 

0.593977 

0.627873 

0.29739 

0.445044 

0.408407 

0.361806 

0.365843 

0.341436 

0.376036 

0.159668 

0.315455 

0.70261 

0.554956 

0.591593 

0.637976 

0.634592 

0.658599 

0.624035 

0.84027 

0.684641 

0.606784 

0.731206 

0.716173 

0.995916 

0.991331 

0.948368 

0.939435 

0.97076 

0.933381 

0.393216 

0.268794 

0.283827 

0.003951 

0.00851 

0.051675 

0.060884 

0.029249 

0.066621 

0.461278 

0.342686 

0.357333 

0.140877 

0.127193 

0.153092 

0.164928 

0.195024 

0.163887 

0.538722 

0.657314 

0.642667 

0.86253 

0.874057 

0.848984 

0.835384 

0.804869 

0.836866 

0.470491 

0.521664 

0.423961 

0.379304 

0.457352 

0.475896 

0.372411 

0.469517 

0.461001 

0.529509 

0.478336 

0.576039 

0.619947 

0.541864 

0.524037 

0.627507 

0.530381 

0.539539 

0.470491 

0.521664 

0.423961 

0.378964 

0.457662 

0.475049 

0.372403 

0.467649 

0.460666 

0.664206 

0.485032 

0.287992 

0.121655 

0.241752 

0.159833 

0.144201 

0.110223 

0.113522 

0.655636 

0.659726 

0.870108 

0.75955 

0.708472 

0.779783 

0.70181 

0.781963 

0.595449 

0.344364 

0.340274 

0.129892 

0.240488 

0.29041 

0.220729 

0.298013 

0.218117 

0.404521 

0.695276 

0.456289 

0.212347 

0.401851 

0.231906 

0.28349 

0.302842 

0.066388 

0.220862 

0.601117 

0.097369 

0.636739 

0.40696 

0.269312 

0.210694 

0.420666 

0.606164 

0.395409 

0.353301 

0.204674 

0.026644 

0.300221 

0.066878 

0.055827 

0.031449 

0.169908 

0.063071 

0.636898 

0.795326 

0.440299 

0.404342 

0.510805 

0.641258 

0.50827 

0.546003 

0.815791 

0.009801 

0 

0.533057 

0.297558 

0.425631 

0.297929 

0.459931 

0.282283 

0.121144 
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16 

17 

18 

19 

20 

21 

22 

23 

24 

0.055581 

0.043707 

0.015816 

0.013637 

0.004381 

0.044006 

0.028698 

0.024597 

0.018179 

0.023539 

0.033061 

0.015753 

0.01171 

0.005436 

0.028533 

0.014351 

0.013468 

0.020985 

0.015472 

0.027504 

0.011738 

0.0078 

0.009748 

0.030663 

0.007281 

0.015052 

0.030575 

0.021241 

0.025434 

0.002453 

0.012238 

0.007957 

0.037986 

0.020479 

0.013681 

0.0281  64 

0.075114 

0.091931 

0.005529 

0.024422 

0.028724 

0.061546 

0.089726 

0.033769 

0.0691  87 

0.14537 

0.117128 

0.178379 

0.160863 

0.208885 

0.149829 

0.163643 

0.16732 

0.153459 

0.012551 

0.029121 

0.01753 

0.016699 

0.000271 

0.008396 

0.002667 

0.003127 

0.007143 

0.035276 

0.008934 

0.027956 

0.027202 

0.029899 

0.020994 

0.025628 

0.038394 

0.019823 

0.01054 

0.019276 

0.024992 

0.017268 

0.025311 

0.005791 

0.003927 

0.000627 

0.000748 

0.001213 

0.03291 

0.063293 

0.023941 

0.067568 

0.061584 

0.033902 

0.044394 

0.07971 

0.107456 

0.086164 

0.08013 

0.108198 

0.079807 

0.069567 

0.103065 

0.087628 

0.058665 

0.128886 

0.030358 

0.038039 

0.140691 

0.093506 

0.077497 

0.057027 

0.170417 

0.12709 

0.06967 

0.061752 

0.095371 

0.077834 

0.101587 

0.076633 

0.062675 

0.0789 

0.081865 

0.084309 

0.084508 

0.100432 

0.099645 

0.10255 

0.081505 

0.102997 

0.091519 

0.076216 

0.046589 

0.081551 

0.088932 

0.055777 

0.054969 

0.055746 

0.042046 

0.05015 

0.052037 

0.042179 

0.064848 

0.040036 

0.013217 

0.016831 

0.038916 

0.00172 

0.018086 

0.036119 

0.079326 

0.075594 

0.110157 

0.08513 

0.092065 

0.082579 

0.068273 

0.08112 

0.074318 

0.023025 

0.0262 

0.018739 

0.040148 

0.015296 

0.029088 

0.04226 

0.026964 

0.026271 

7912 

7934 

7532 

7460 

7112 

7416 

7444 

7170 

7306 

0.200778 

0.237747 

0.059803 

0.073326 

0.073363 

0.217352 

0.220457 

0.101812 

0.196642 

0.593396 

0.469276 

0.624887 

0.673608 

0.708789 

0.555259 

0.557199 

0.681669 

0.60327 

0.205376 

0.294749 

0.311869 

0.252507 

0.217833 

0.226186 

0.222153 

0.216336 

0.200661 

0.518829 

0.505659 

0.866568 

0.500058 

0.499999 

0.539825 

0.500111 

0.533179 

0.504861 

0.480024 

0.494201 

0.133352 

0.499758 

0.499999 

0.457906 

0.5 

0.466058 

0.494992 

0.266131 

0.312137 

0.312999 

0.263645 

0.295073 

0.284932 

0.298589 

0.25 

0.266253 

0.259423 

0.311998 

0.31286 

0.263534 

0.285528 

0.284777 

0.297813 

0.25 

0.262446 

0.217202 

0.06469 

0.060873 

0.208861 

0.131924 

0.139569 

0.102347 

0.25 

0.210668 

0.259477 

0.31203 

0.312926 

0.263562 

0.287965 

0.284826 

0.297924 

0.249999 

0.262495 

0.501634 

0.654446 

0.584451 

0.50381 

0.507139 

0.507396 

0.50132 

0.500715 

0.499994 

0.498361 

0.345394 

0.413326 

0.496299 

0.492861 

0.493075 

0.497575 

0.499376 

0.499987 

0.533557 

0.363563 

0.30519 

0.371925 

0.119424 

0.375057 

0.26029 

0.45311 

0.183117 

0.466434 

0.636451 

0.694404 

0.627093 

0.87628 

0.623966 

0.743986 

0.54689 

0.818991 

0.220397 

0.216466 

0.307571 

0.333966 

0.216936 

0.381664 

0.185785 

0.288316 

0.289349 

0.77937 

0.782431 

0.693252 

0.666455 

0.783073 

0.61839 

0.814005 

0.711683 

0.710808 

0.920521 

0.799625 

0.91201 

0.905795 

0.99876 

0.94702 

0.983586 

0.981823 

0.955404 

0.079478 

0.198809 

0.089626 

0.094028 

0.001294 

0.053068 

0.01603 

0.018349 

0.044471 

0.229152 

0.061018 

0.142995 

0.153234 

0.147738 

0.13277 

0.154443 

0.225293 

0.125202 

0.771219 

0.940986 

0.858567 

0.844387 

0.853194 

0.86742 

0.847651 

0.776893 

0.876852 

0.452485 

0.421721 

0.487722 

0.438411 

0.497717 

0.48455 

0.377556 

0.462984 

0.51697 

0.54756 

0.57713 

0.513606 

0.56126 

0.502435 

0.515359 

0.620451 

0.53703 

0.481294 

0.452214 

0.422771 

0.487907 

0.438446 

0.497586 

0.485737 

0.377535 

0.462968 

0.51866 

0.409416 

0.336178 

0.219651 

0.103893 

0.152355 

0.339091 

0.015448 

0.166964 

0.360007 

0.775039 

0.742543 

0.85457 

0.67931 

0.857523 

0.740439 

0.617648 

0.750879 

0.740749 

0.224964 

0.257358 

0.145371 

0.320363 

0.142477 

0.26082 

0.382312 

0.249589 

0.261856 

0.484451 

0.432096 

0.437746 

0.50929 

0.169123 

0.389732 

0.17496 

0.453668 

0.258102 

0.318429 

0.359462 

0.326187 

0.339234 

0.303266 

0.418825 

0.088762 

0.507002 

0.376045 

0.088777 

0.139909 

0.148895 

0.115202 

0.146587 

0.042221 

0.027915 

0.004738 

0.00539 

0.905113 

0.625408 

0.477386 

0.721827 

0.462197 

0.507197 

0.732725 

0.661867 

0.422569 

0.010214 

0.23887 

0.377078 

0.159719 

0.391317 

0.448989 

0.241023 

0.335314 

0.574162 
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25 

26 

27 

28 

29 

30 

31 

32 

33 

0.019101 

0.012853 

0.030557 

0.005685 

0.042177 

0.023096 

0.063212 

0.052603 

0.033707 

0.014811 

0.010386 

0.035306 

0.00376 

0.026735 

0.017483 

0.015997 

0.039928 

0.048239 

0.022726 

0.012472 

0.039695 

0.007008 

0.042615 

0.020408 

0.022346 

0.0433 

0.052175 

0.036557 

0.012131 

0.023773 

0.003846 

0.06071 

0.042784 

0.053303 

0.044074 

0.039195 

0.056972 

0.030707 

0.069993 

0.010065 

0.052324 

0.052745 

0.050465 

0.100103 

0.086151 

0.15899 

0.165425 

0.103745 

0.234225 

0.087988 

0.165686 

0.142856 

0.055346 

0.082014 

0.006963 

0.008227 

0.05249 

0.004899 

0.050894 

0.010569 

0.003226 

0.049258 

0.04264 

0.019291 

0.03032 

0.06356 

0.013423 

0.026896 

0.02941 

0.030898 

0.0033 

0.032007 

0.001672 

0.017241 

0 

0.005032 

0 

0.005818 

0.022765 

0.013817 

0.033031 

0.070449 

0.02763 

0.031341 

0.198065 

0.032468 

0.049297 

0.008412 

0.018477 

0.003374 

0.073994 

0.097681 

0.061333 

0.019659 

0.079519 

0.092291 

0.084007 

0.069129 

0.056242 

0.123232 

0.156663 

0.156235 

0.217008 

0.097911 

0.088059 

0.146083 

0.094285 

0.124654 

0.078575 

0.083507 

0.066021 

0.021048 

0.037155 

0.089282 

0.060443 

0.048646 

0.06098 

0.087071 

0.089824 

0.090214 

0.216982 

0.101727 

0.086909 

0.08564 

0.055958 

0.063674 

0.045305 

0.05759 

0.039623 

0.001434 

0.04187 

0.049082 

0.043329 

0.086165 

0.061352 

0.027977 

0.017429 

0.044167 

0.001122 

0.045413 

0.037002 

0.075033 

0.099823 

0.096288 

0.079362 

0.0837 

0.062321 

0.014872 

0.043588 

0.082095 

0.078352 

0.064731 

0.072267 

0.015502 

0.035038 

0.031445 

0.000836 

0.042628 

0.015058 

0.02506 

0.044591 

0.053012 

7152 

7374 

8452 

6538 

8344 

7296 

8074 

8460 

8808 

0.187647 

0.085509 

0.187531 

0.027921 

0.24272 

0.191151 

0.207537 

0.295946 

0.250692 

0.62137 

0.676672 

0.624938 

0.935583 

0.514559 

0.616646 

0.58433 

0.408097 

0.498617 

0.189993 

0.238267 

0.187531 

0.031914 

0.24272 

0.193679 

0.208133 

0.295957 

0.250692 

0.500192 

0.500009 

0.5 

0.50286 

0.534303 

0.500041 

0.5 

0.512816 

0.5 

0.498568 

0.499996 

0.5 

0.497872 

0.465697 

0.499986 

0.5 

0.487174 

0.5 

0.267128 

0.256597 

0.25 

0.255758 

0.269906 

0.285717 

0.25 

0.256323 

0.25 

0.266865 

0.255752 

0.25 

0.252835 

0.269906 

0.285637 

0.25 

0.256323 

0.25 

0.198323 

0.23152 

0.25 

0.23195 

0.190281 

0.142804 

0.25 

0.231035 

0.25 

0.267131 

0.256212 

0.25 

0.25446 

0.269906 

0.28566 

0.25 

0.256322 

0.25 

0.500673 

0.501592 

0.5 

0.50708 

0.644792 

0.500003 

0.50314 

0.626041 

0.500269 

0.499109 

0.498385 

0.5 

0.492144 

0.355208 

0.499994 

0.49686 

0.369384 

0.499731 

0.203508 

0.300623 

0.325889 

0.404945 

0.325226 

0.241633 

0.609163 

0.346607 

0.268916 

0.794606 

0.699325 

0.674111 

0.595912 

0.674774 

0.758413 

0.390837 

0.653394 

0.731084 

0.390755 

0.28374 

0.253535 

0.276651 

0.537097 

0.447664 

0.513676 

0.305692 

0.312694 

0.608972 

0.71821 

0.746465 

0.724056 

0.462903 

0.551885 

0.486324 

0.694308 

0.687306 

0.957858 

0.952735 

0.664031 

0.978862 

0.633544 

0.940402 

0.977916 

0.529099 

0.657934 

0.041947 

0.047381 

0.335969 

0.020472 

0.366456 

0.059989 

0.022084 

0.470901 

0.342066 

0.116334 

0.175199 

0.406824 

0.056743 

0.193659 

0.16697 

0.211511 

0.031543 

0.256768 

0.884234 

0.824748 

0.593176 

0.939925 

0.806341 

0.833022 

0.788489 

0.969574 

0.743232 

0.473382 

0.481667 

0.422573 

0.088411 

0.267531 

0.50685 

0.413757 

0.465049 

0.489197 

0.524563 

0.518098 

0.577427 

0.911426 

0.732469 

0.493374 

0.586243 

0.534951 

0.510803 

0.476268 

0.48187 

0.422573 

0.088592 

0.267531 

0.506838 

0.413757 

0.465048 

0.489197 

0.294104 

0.145834 

0.471036 

0.069347 

0.290172 

0.382098 

0.716511 

0.538764 

0.767763 

0.836905 

0.70485 

0.664645 

0.946883 

0.505564 

0.847757 

0.757667 

0.592118 

0.576849 

0.163473 

0.295065 

0.335355 

0.053216 

0.494436 

0.155501 

0.242333 

0.407883 

0.423151 

0.198593 

0.347357 

0.558571 

0.449381 

0.30551 

0.241169 

0.39444 

0.40265 

0.526406 

0.30472 

0.417135 

0.628 

0.83757 

0.486974 

0.281523 

0.550989 

0.436659 

0.569357 

0.011401 

0.120794 

0 

0.022634 

0 

0.039652 

0.197639 

0.136231 

0.356528 

0.504645 

0.684375 

0.661812 

0.088419 

0.710074 

0.629001 

0.729327 

0.681597 

0.607059 

0.480472 

0.193581 

0.338188 

0.890828 

0.289926 

0.335983 

0.073034 

0.182175 

0.036413 
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34 

35 

36 

37 

38 

39 

40 

41 

42 

0.068413 

0.024624 

0.029754 

0.012686 

0.035493 

0.041048 

0.043239 

0.014321 

0.019589 

0.0332 

0.010688 

0.030903 

0.017964 

0.021687 

0.022559 

0.018212 

0.017087 

0.017714 

0.033219 

0.007194 

0.011926 

0.013191 

0.019773 

0.012925 

0.040148 

0.012175 

0.020236 

0.043419 

0.018311 

0.027639 

0.015081 

0.022189 

0.015465 

0.007779 

0.01811 

0.026077 

0.033951 

0.066702 

0.052877 

0.027096 

0.050936 

0.073933 

0.11163 

0.034869 

0.067939 

0.170443 

0.156759 

0.13327 

0.17797 

0.154294 

0.145111 

0.113157 

0.181361 

0.182608 

0 

0.008054 

0.010704 

0.016477 

0.020038 

0.015511 

0.044769 

0.021944 

2E-05 

0.042865 

0.02473 

0.022391 

0.024869 

0.02603 

0.020221 

0.015234 

0.033108 

0.027621 

0.03409 

0.000544 

0.008812 

0.013033 

0.007276 

0.001175 

0.005657 

0.026642 

0.005979 

0.024216 

0.08291 

0.027118 

0.054319 

0.013632 

0.041633 

0.028549 

0.024787 

0.0597 

0.069273 

0.053936 

0.083486 

0.103048 

0.126707 

0.096763 

0.092647 

0.104341 

0.080637 

0.031152 

0.162088 

0.117766 

0.045743 

0.060831 

0.07493 

0.063878 

0.040836 

0.046992 

0.067368 

0.077652 

0.074175 

0.082876 

0.077406 

0.083366 

0.046671 

0.088322 

0.080281 

0.103075 

0.083565 

0.067857 

0.111484 

0.096173 

0.077438 

0.095417 

0.100621 

0.095705 

0.045209 

0.042523 

0.083195 

0.064605 

0.062084 

0.069441 

0.040567 

0.060833 

0.052281 

0.093033 

0.021035 

0.041145 

0.020509 

0.026858 

0.032473 

0.085941 

0.025121 

0.018048 

0.101336 

0.079177 

0.079992 

0.096232 

0.084606 

0.09049 

0.083193 

0.08708 

0.089292 

0.013043 

0.008996 

0.045358 

0.037532 

0.04329 

0.042491 

0.040312 

0.043277 

0.02336 

8320 

7026 

7802 

7276 

7570 

7474 

8094 

7424 

7132 

0.228759 

0.171584 

0.16923 

0.085135 

0.149346 

0.178402 

0.24701 

0.11731 

0.193771 

0.542482 

0.650875 

0.545068 

0.629249 

0.583521 

0.55532 

0.505979 

0.621998 

0.579018 

0.228759 

0.176911 

0.285818 

0.285487 

0.267116 

0.266567 

0.24701 

0.26066 

0.227211 

0.661782 

0.504464 

0.523785 

0.504501 

0.508253 

0.500008 

0.516587 

0.550385 

0.514289 

0.338218 

0.495454 

0.475778 

0.495273 

0.489636 

0.500006 

0.483413 

0.451467 

0.485629 

0.314192 

0.253162 

0.26412 

0.308896 

0.298753 

0.28924 

0.312121 

0.326806 

0.315413 

0.314192 

0.249062 

0.259706 

0.308643 

0.297401 

0.287638 

0.280817 

0.302427 

0.300251 

0.057425 

0.24903 

0.216056 

0.072695 

0.104248 

0.134931 

0.126246 

0.065653 

0.081159 

0.314192 

0.249066 

0.2605 

0.308843 

0.297473 

0.288595 

0.280817 

0.303881 

0.303921 

0.5 

0.501799 

0.558087 

0.531599 

0.521189 

0.500627 

0.5 

0.499998 

0.504271 

0.5 

0.498317 

0.438608 

0.468381 

0.479038 

0.4994 

0.5 

0.499999 

0.495744 

0.451902 

0.284483 

0.335676 

0.295372 

0.46759 

0.460171 

0.338855 

0.221812 

0.196566 

0.548098 

0.714531 

0.663358 

0.704603 

0.532439 

0.539842 

0.661145 

0.778354 

0.803468 

0.561187 

0.215391 

0.343271 

0.357654 

0.303445 

0.173376 

0.065142 

0.341946 

0.27712 

0.438813 

0.784621 

0.656727 

0.642607 

0.696563 

0.828827 

0.934858 

0.658385 

0.721981 

1 

0.951341 

0.925717 

0.915614 

0.885076 

0.903436 

0.716519 

0.892203 

0.99988 

0 

0.048881 

0.074352 

0.084771 

0.114942 

0.096569 

0.283481 

0.107954 

0.000109 

0.251488 

0.152552 

0.158179 

0.128051 

0.149993 

0.126595 

0.107218 

0.176006 

0.158879 

0.748512 

0.847467 

0.841837 

0.872184 

0.850121 

0.875901 

0.892782 

0.823699 

0.84148 

0.395252 

0.479005 

0.522395 

0.426451 

0.445936 

0.520183 

0.328467 

0.467279 

0.456205 

0.604748 

0.51548 

0.477896 

0.573656 

0.55405 

0.483194 

0.671533 

0.532348 

0.54385 

0.395252 

0.479 

0.521561 

0.425689 

0.445949 

0.520353 

0.328467 

0.466762 

0.456297 

0.813371 

0.236956 

0.257945 

0.135139 

0.19292 

0.243331 

0.695849 

0.192748 

0.157518 

0.885964 

0.898128 

0.638085 

0.719672 

0.661199 

0.679745 

0.673597 

0.66815 

0.792729 

0.114036 

0.102039 

0.361817 

0.280682 

0.33831 

0.319186 

0.326403 

0.332055 

0.207393 

0.400111 

0.172805 

0.525555 

0.593576 

0.536608 

0.468472 

0.215869 

0.340002 

0.221231 

0.400344 

0.116318 

0.202825 

0.435878 

0.489261 

0.268393 

0.475896 

0.242265 

0.252733 

0.267204 

0.003959 

0.073943 

0.076942 

0.049319 

0.008402 

0.044595 

0.171945 

0.040873 

0.542982 

0.3926 

0.70057 

0.608347 

0.858856 

0.691613 

0.730347 

0.673403 

0.551206 

0.189814 

0.603506 

0.227558 

0.320676 

0.092401 

0.297573 

0.225058 

0.159972 

0.408087 
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43  44 

45 

46 

47 

48 

49  50 

0.045339  0.023279 

0.030762 

0.034218 

0.026374 

0.038239 

0.03401  0.021475 

0.029807  0.012094 

0.024259 

0.021468 

0.005942 

0.042379 

0.02347  0.007783 

0.029211  0.005799 

0.024921 

0.029482 

0.022601 

0.032706 

0.024884  0.01141 

0.02205  0.024306 

0.021742 

0.022231 

0.031859 

0.024676 

0.022698  0.016151 

0.081485  0.046626 

0.068501 

0.061918 

0.0451 

0.086111 

0.070102  0.052084 

0.144456  0.138591 

0.171799 

0.151129 

0.143979 

0.117932 

0.143367  0.181464 

0.008791  0.012491 

0.004043 

0.00915 

0.015241 

0.017799 

0.015549  0.001666 

0.025978  0.025831 

0.021954 

0.020892 

0.013136 

0.028874 

0.02927  0.024743 

0.000612  0.001522 

0.011641 

0.002128 

0.000454 

0.00072 

0.008546  0.001233 

0.046344  0.033141 

0.044685 

0.054057 

0.054417 

0.033209 

0.034122  0.058942 

0.079641  0.084002 

0.097537 

0.079264 

0.0912 

0.072808 

0.086342  0.066501 

0.089684  0.14082 

0.047828 

0.12388 

0.150621 

0.135597 

0.063202  0.151256 

0.075733  0.070118 

0.076711 

0.07402 

0.068828 

0.056458 

0.078737  0.071926 

0.076921  0.080237 

0.098913 

0.082287 

0.090244 

0.079268 

0.079544  0.0799 

0.060079  0.048863 

0.054577 

0.057497 

0.047016 

0.047582 

0.067729  0.044102 

0.057025  0.02023 

0.028503 

0.030757 

0.012684 

0.049733 

0.032094  0.016287 

0.078333  0.076871 

0.091718 

0.071154 

0.075273 

0.06079 

0.082092  0.07029 

0.039637  0.026848 

0.02717 

0.041516 

0.033087 

0.053579 

0.054061  0.01822 

7598  7272 

7350 

7466 

7336 

7988 

7586  7058 

_ 1 

0.208676  0.205406 

0.180499 

0.168502 

0.155261 

0.228586 

0.189015  0.181238 

0.548145  0.586353 

0.574434 

0.596447 

0.627334 

0.542669 

0.538679  0.636973 

0.239065  0.208533 

0.244158 

0.234122 

0.217049 

0.228757 

0.272306  0.181594 

0.50383  0.657573 

0.500001 

0.500394 

0.504337 

0.515353 

0.509014  0.620111 

0.496151  0.345326 

0.499929 

0.499607 

0.495663 

0.484642 

0.490986  0.376614 

0.279575  0.257663 

0.305754 

0.268732 

0.253788 

0.250127 

0.295009  0.286794 

0.278345  0.246215 

0.305456 

0.261474 

0.252639 

0.249883 

0.293832  0.237461 

0.163613  0.240163 

0.08326 

0.207696 

0.240096 

0.249871 

0.117327  0.237461 

0.278492  0.256385 

0.305747 

0.262087 

0.253568 

0.250121 

0.293832  0.238507 

0.506562  0.502542 

0.51321 

0.518736 

0.500645 

0.499991 

0.500001  0.513477 

0.493454  0.497436 

0.48694 

0.481266 

0.499213 

0.499993 

0.5  0.48688 

0.431234  0.172346 

0.34085 

0.405831 

0.336821 

0.324599 

0.35349  0.19108 

0.568761  0.827247 

0.659966 

0.594919 

0.663157 

0.675269 

0.64651  0.807983 

0.212974  0.342664 

0.240945 

0.264074 

0.413983 

0.222742 

0.244584  0.236615 

0.787025  0.657338 

0.759125 

0.735498 

0.586042 

0.777303 

0.75538  0.763065 

0.942631  0.917323 

0.978157 

0.942879 

0.904332 

0.868833 

0.90216  0.993342 

0.057363  0.082677 

0.023021 

0.057087 

0.095729 

0.131128 

0.097846  0.00912 

0.170267  0.178925 

0.125118 

0.133963 

0.082885 

0.212929 

0.184926  0.163581 

0.829739  0.821048 

0.875411 

0.868214 

0.921443 

0.78708 

0.815074  0.836418 

0.49611  0.466417 

0.436773 

0.473514 

0.432688 

0.415949 

0.497451  0.473437 

0.503891  0.533734 

0.563186 

0.526398 

0.567315 

0.584002 

0.50255  0.525923 

0.496109  0.466262 

0.435556 

0.473433 

0.432673 

0.416017 

0.49745  0.472972 

0.470891  0.193041 

0.22747 

0.253255 

0.116724 

0.434816 

0.235721  0.174671 

0.664019  0.741059 

0.771452 

0.631502 

0.694698 

0.531491 

0.60294  0.795005 

0.336  0.258821 

0.228525 

0.368459 

0.305359 

0.468444 

0.397058  0.206079 

0.498463  0.108234 

0.407298 

0.427974 

0.114428 

0.532748 

0.37732  0.085707 

0.4885  0.051896 

0.418404 

0.587747 

0.435247 

0.411152 

0.400061  0.125653 

0.004831  0.012837 

0.075788 

0.015717 

0.003109 

0.006742 

0.066244  0.009743 

0.629095  0.708678 

0.634995 

0.585399 

0.624492 

0.682165 

0.669264  0.525645 

0.366076  0.279594 

0.290909 

0.399233 

0.372619 

0.311142 

0.264491  0.465898 
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Appendix  D 


This  appendix  contains  the  data  on  the  measures  for  the  second  MILCON 
example: 


Force  Structure 

Consolidation 

Footprint 

Reduction 

Joint  Use 

Payback 

Deficit  IRR 

Restoration  and 

Modernization 

IRR 

Design  Build 

Years  to  IOC 

Mission  Panel 

Points 

Installation  CC 
Priority  Points 

Corrects  ATFP 
Deficiency 
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0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

0.06 

0.33 

0.00 

0.00 

0.00 

0.28 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

0.04 

0.14 

0.00 

0.00 

0.00 

0.37 

0.00 

1.00 

0.80 

0.00 

1.00 

0.70 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.33 

0.00 

0.00 

0.00 

0.70 

0.00 

1.00 

1.00 

0.00 

1.00 

0.70 

0.00 

0.00 

0.00 

1.00 

1.00 

0.00 

0.11 

0.50 

0.00 

0.00 

1.00 

0.31 

0.00 

0.00 

0.30 

0.00 

1.00 

0.70 

1.00 

0.00 

0.00 

0.30 

0.00 

1.00 

0.01 

0.50 

0.00 

0.00 

0.00 

0.35 

0.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.80 

0.00 

0.00 

0.01 

0.25 

0.00 

0.00 

0.00 

0.51 

0.00 

1.00 

0.80 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.80 

1.00 

0.00 

0.17 

0.33 

0.00 

0.00 

0.00 

0.49 

0.00 

1.00 

0.30 

0.00 

0.00 

0.70 

0.00 

0.00 

0.00 

0.30 

0.00 

0.00 

0.06 

0.33 

0.00 

0.00 

0.00 

0.69 

0.00 

1.00 

1.00 

0.00 

0.00 

0.70 

0.00 

0.00 

0.00 

0.80 

1.00 

0.00 

0.02 

0.33 

0.00 

0.00 

0.00 

0.31 

0.00 

1.00 

0.30 

0.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.80 

0.00 

0.00 

0.02 

0.17 

0.00 

0.00 

0.00 

0.33 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.30 

0.00 

0.00 

0.04 

1.00 

0.00 

1.00 

0.00 

0.62 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

1.00 

0.00 

0.00 

0.04 

0.33 

0.00 

0.00 

0.00 

0.32 

0.00 

1.00 

0.50 

0.00 

0.00 

0.70 

0.00 

0.00 

0.00 

0.30 

0.00 

0.00 

0.03 

0.33 

0.00 

0.00 

0.00 

0.72 

0.00 

1.00 

0.80 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.07 

0.33 

1.00 

0.00 

0.00 

1.00 

0.00 

1.00 

0.50 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.30 

0.00 

0.00 

0.17 

1.00 

0.00 

0.00 

0.00 

0.22 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.30 

0.00 

0.00 

0.05 

0.50 

0.00 

0.00 

0.00 

0.87 

0.00 

1.00 

0.50 

0.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.30 

0.00 

0.00 

0.04 

1.00 

0.00 

0.00 

0.00 

0.57 

0.00 

1.00 

0.50 

0.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.80 

0.00 

0.00 

0.06 

0.50 

0.00 

0.00 

0.00 

0.27 

0.00 

1.00 

0.50 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.06 

0.25 

1.00 

0.00 

0.00 

1.00 

0.00 

1.00 

0.50 

0.00 

1.00 

0.70 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.14 

0.33 

0.00 

1.00 

0.00 

0.64 

0.00 

0.00 

0.50 
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Appendix  E 


This  appendix  contains  the  output  from  the  50  runs  on  the  MILCON  example  #  2: 


Known 

1 

2 

3 

4 

5 

6 

Face  Structure 

0.072 

0.096292 

0.084003 

0.088433 

0.052029 

0.049109 

0.080281 

Consol  idatbn 

0.0168 

0.02149 

0.020733 

0.021197 

0.021591 

0.017689 

0.018887 

Foorprint  Reduction 

0.0312 

0.038742 

0.036681 

0.035352 

0.040162 

0.037419 

0.035594 

Joint  Use 

0.04 

0.050151 

0.040359 

0.053394 

0.04502 

0.039135 

0.064375 

Payback 

0.04 

0.058313 

0.056889 

0.074582 

0.095567 

0.048323 

0.077083 

Deficit  IRR 

0.1125 

0.060537 

0.065302 

0.009342 

0.033146 

0.005258 

0.052117 

Rest.  &  Modern.  IRR 

0.1125 

0.139992 

0.126467 

0.131821 

0.142463 

0.137845 

0.129116 

Design  Build 

0.0075 

0.005305 

0.005888 

0.008792 

0.00648 

0.013079 

0.004508 

Years  to  lOC/Need  date 

0.00675 

0.000319 

0.002578 

0.000256 

0.025162 

0.061351 

0.002739 

Mission  Pannel  Points 

0.02025 

0.00346 

0.001533 

7.85E-05 

0.022813 

0.001203 

0.000972 

Installation  CC's  Prbrity 

0.0405 

0.051218 

0.043012 

0.045828 

0.050815 

0.056504 

0.050334 

Corrects  ATFP  Deficiency 

0.025 

0.048352 

0.036196 

0.040035 

0.0294 

0.086622 

0.04209 

Direct  Operational  Support 

0.1225 

0.082949 

0.083808 

0.07501 

0.080151 

0.100034 

0.091531 

Warfighting  Enablers 

0.0525 

0.026494 

0.058388 

0.059311 

0.02485 

0.039727 

0.03299 

Facility  Age 

0.126 

0.159821 

0.145146 

0.146653 

0.161857 

0.146552 

0.145938 

Eliminates  Safety  Violation 

0.054 

0.040121 

0.17382 

0.104505 

0.130468 

0.065233 

0.048541 

Support  Facility 

0.096 

0.051723 

0.055344 

0.044334 

0.046911 

0.06925 

0.062727 

Base  Populatbn 

0.024 

0.030612 

0.020503 

0.022052 

0.031496 

0.038001 

0.033593 

Error 

870 

1052 

1250 

1022 

1486 

1162 

Difference  in  weights 

0.368075 

0.365737 

0.422832 

0.479248 

0.4364 

0.325164 

Efficiencies 

0.29175 

0.252279 

0.296965 

0.292054 

0.183343 

0.293769 

op  suppt 

0.416281 

0.424898 

0.372114 

0.412875 

0.505048 

0.407914 

QOL 

0.294803 

0.321968 

0.329254 

0.292373 

0.311902 

0.296034 

operational 

0.627298 

0.614297 

0.568014 

0.516012 

0.523854 

0.512888 

resources 

0.372659 

0.385634 

0.431436 

0.481961 

0.476846 

0.484388 

readiness 

0.48179 

0.45045 

0.37847 

0.423458 

0.283002 

0.444268 

responsiveness 

0.144174 

0.124508 

0.147297 

0.253813 

0.26299 

0.143061 

secu  rity 

0.116152 

0.085187 

0.107588 

0.071209 

0.171512 

0.103183 

missions 

0.262809 

0.33439 

0.361003 

0.254391 

0.276983 

0.304619 

work  QOL 

0.715363 

0.764543 

0.798181 

0.728942 

0.657642 

0.667506 

sense  of  community 

0.279379 

0.235421 

0.200713 

0.268409 

0.344164 

0.326546 

right  place 

0.526143 

0.542042 

0.524265 

0.345239 

0.511316 

0.532821 

right  size 

0.473924 

0.456636 

0.474753 

0.652193 

0.488539 

0.467266 

joint  use 

0.461276 

0.414847 

0.416748 

0.319842 

0.447632 

0.452393 

econ 

0.536341 

0.584756 

0.582118 

0.678941 

0.552725 

0.541697 

reduce  deficit 

0.301839 

0.341189 

0.066333 

0.189582 

0.036787 

0.287587 

restore  and  mondernize 

0.698006 

0.660762 

0.936001 

0.814839 

0.964425 

0.71247 

ability  to  execute 

0.088399 

0.111298 

0.160405 

0.061835 

0.098468 

0.077255 

mission  timing 

0.915992 

0.890764 

0.840447 

0.941272 

0.898584 

0.92281 

combat  capability 

0.758197 

0.589859 

0.558387 

0.763114 

0.715093 

0.736618 

mission  support 

0.242167 

0.410951 

0.44152 

0.236592 

0.283989 

0.265497 

modern  facilities 

0.757835 

0.589646 

0.558032 

0.75945 

0.714468 

0.738537 

safe  facil 

0.190245 

0.70613 

0.397655 

0.612171 

0.318024 

0.245648 

support  facilities 

0.627995 

0.730146 

0.670861 

0.597774 

0.645115 

0.648883 

promotes  community 

0.37168 

0.270499 

0.333684 

0.401354 

0.354006 

0.347506 

Consol  idatbn 

0.247768 

0.29297 

0.264697 

0.219668 

0.376998 

0.268267 

Foorprint  Reduction 

0.446666 

0.518329 

0.441451 

0.40862 

0.797481 

0.505569 

Years  to  lOC/Need  date 

0.005805 

0.054706 

0.005558 

0.255096 

0.514032 

0.050853 

Installatbn  CC's  Prbrity 

0.931664 

0.912738 

0.994841 

0.515159 

0.473419 

0.934669 

Mission  Panel  Points 

0.062935 

0.032523 

0.001703 

0.231275 

0.010078 

0.018054 
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7 

8 

9 

10 

11 

12 

13 

14 

15 

0.06179 

0.085585 

0.058382 

0.039258 

0.073342 

0.061236 

0.082655 

0.091257 

0.050903 

0.018573 

0.01979 

0.017852 

0.016843 

0.018249 

0.01898 

0.019319 

0.020303 

0.018962 

0.03398 

0.033876 

0.034677 

0.030502 

0.035206 

0.035512 

0.034643 

0.03748 

0.034571 

0.062296 

0.055187 

0.042725 

0.043604 

0.046342 

0.059853 

0.046093 

0.048691 

0.04207 

0.063145 

0.07057 

0.059508 

0.082057 

0.050362 

0.081839 

0.078172 

0.076126 

0.050046 

0.055206 

0.075444 

0.027122 

0.094739 

0.065965 

0.054385 

0.025967 

0.04767 

0.06375 

0.122965 

0.122776 

0.12231 

0.10947 

0.127374 

0.129542 

0.12339 

0.133586 

0.124742 

0.005375 

0.004831 

0.006058 

0.011955 

0.008483 

0.006408 

0.00704 

0.006592 

0.0091  03 

0.001707 

0.000183 

0.030437 

0.000396 

0.002316 

0.001561 

0.002156 

0.001994 

0.029311 

0.001904 

0.000467 

0.021455 

0.000216 

0.001844 

0.001071 

0.018111 

0.000723 

0.012498 

0.045182 

0.043296 

0.046254 

0.035762 

0.045236 

0.044927 

0.043696 

0.049113 

0.043947 

0.042311 

0.048283 

0.087262 

0.038738 

0.045083 

0.042694 

0.025726 

0.04972 

0.081432 

0.089222 

0.10806 

0.07972 

0.061939 

0.080171 

0.083439 

0.088534 

0.094336 

0.051127 

0.049679 

0.031043 

0.050428 

0.07457 

0.064644 

0.046946 

0.053194 

0.032322 

0.045689 

0.136719 

0.137781 

0.133756 

0.117352 

0.141831 

0.142643 

0.142156 

0.150311 

0.138614 

0.148811 

0.064809 

0.129736 

0.16025 

0.14275 

0.083398 

0.117805 

0.114694 

0.123344 

0.061624 

0.08113 

0.050887 

0.037272 

0.051225 

0.054055 

0.059789 

0.063561 

0.022503 

0.02709 

0.026879 

0.029768 

0.018188 

0.025905 

0.023455 

0.022213 

0.03016 

0.024507 

1166 

1024 

1382 

1646 

924 

1360 

902 

986 

1278 

0.354566 

0.243673 

0.409143 

0.411323 

0.339285 

0.334831 

0.322041 

0.390243 

0.425677 

0.28232 

0.278189 

0.223468 

0.256463 

0.224749 

0.28822 

0.299287 

0.287545 

0.226224 

0.415148 

0.434869 

0.472728 

0.430362 

0.444564 

0.410983 

0.386626 

0.416485 

0.464843 

0.30141 

0.28534 

0.299216 

0.313515 

0.334641 

0.300538 

0.308743 

0.295262 

0.309302 

0.548641 

0.547319 

0.53668 

0.508375 

0.56766 

0.504323 

0.581706 

0.564853 

0.590928 

0.446317 

0.452064 

0.457826 

0.491283 

0.428328 

0.491656 

0.416556 

0.433917 

0.408252 

0.429316 

0.455795 

0.316133 

0.474479 

0.435187 

0.447627 

0.385233 

0.435205 

0.405438 

0.130015 

0.112151 

0.22106 

0.112355 

0.130182 

0.131103 

0.182043 

0.13999 

0.204974 

0.101918 

0.111028 

0.184593 

0.090014 

0.101409 

0.103882 

0.066541 

0.11938 

0.1751  82 

0.33462 

0.319865 

0.275438 

0.317185 

0.327267 

0.317529 

0.366542 

0.304246 

0.208292 

0.706194 

0.621595 

0.729025 

0.824922 

0.769878 

0.743562 

0.736941 

0.683445 

0.84868 

0.294245 

0.378382 

0.269579 

0.176778 

0.229922 

0.257128 

0.264525 

0.317385 

0.151273 

0.398922 

0.562104 

0.486797 

0.301104 

0.574864 

0.421286 

0.474762 

0.561854 

0.380775 

0.607099 

0.441894 

0.513379 

0.698549 

0.430176 

0.584519 

0.526336 

0.438204 

0.6221  86 

0.494398 

0.438833 

0.417602 

0.346077 

0.481389 

0.422375 

0.369721 

0.390243 

0.455516 

0.501134 

0.56115 

0.581645 

0.651266 

0.523153 

0.577532 

0.627036 

0.610131 

0.541879 

0.309746 

0.380624 

0.181488 

0.463959 

0.34096 

0.295621 

0.174344 

0.262998 

0.338259 

0.689927 

0.619422 

0.818428 

0.536098 

0.658372 

0.704158 

0.828451 

0.737001 

0.661886 

0.099579 

0.099046 

0.057975 

0.247244 

0.146581 

0.11892 

0.100021 

0.113062 

0.095538 

0.901178 

0.89997 

0.942114 

0.751103 

0.848986 

0.881196 

0.905458 

0.888544 

0.901778 

0.64227 

0.776853 

0.612253 

0.453749 

0.551041 

0.639385 

0.624735 

0.744481 

0.528048 

0.357619 

0.223171 

0.387292 

0.546283 

0.444316 

0.359739 

0.375363 

0.255075 

0.471882 

0.642316 

0.776818 

0.61318 

0.453752 

0.550514 

0.638312 

0.624792 

0.744867 

0.528055 

0.699123 

0.365395 

0.59475 

0.619622 

0.554082 

0.373196 

0.517766 

0.568367 

0.469884 

0.694839 

0.751429 

0.630859 

0.672504 

0.66576 

0.699494 

0.732076 

0.678262 

0.480941 

0.305454 

0.248957 

0.369047 

0.328175 

0.336685 

0.303517 

0.271983 

0.321837 

0.523772 

0.19751 

0.294131 

0.289946 

0.184936 

0.332509 

0.223389 

0.210832 

0.285263 

0.227971 

0.361354 

0.503493 

0.56322 

0.334911 

0.641487 

0.417965 

0.378061 

0.526602 

0.415645 

0.035095 

0.004165 

0.309152 

0.010909 

0.047137 

0.032884 

0.03383 

0.038484 

0.34113 

0.92888 

0.986402 

0.469813 

0.984679 

0.920664 

0.946228 

0.685663 

0.948037 

0.511476 

0.039151 

0.010649 

0.21792 

0.005957 

0.037537 

0.022561 

0.284193 

0.013964 

0.145463 
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16 

17 

18 

19 

20 

21 

22 

23 

24 

0.060172 

0.068823 

0.100057 

0.073692 

0.06019 

0.069987 

0.075964 

0.070563 

0.083709 

0.018113 

0.016707 

0.021578 

0.022639 

0.018313 

0.019053 

0.019324 

0.018215 

0.020625 

0.034397 

0.030989 

0.038663 

0.034582 

0.036702 

0.03475 

0.037573 

0.033804 

0.037266 

0.044459 

0.046427 

0.018583 

0.054421 

0.046163 

0.048719 

0.050678 

0.043641 

0.05269 

0.081639 

0.064537 

0.081044 

0.066681 

0.056503 

0.074673 

0.087242 

0.059627 

0.063111 

0.030198 

0.042975 

0.051872 

0.025977 

0.026313 

0.05606 

0.027509 

0.051001 

0.060425 

0.127887 

0.112808 

0.142583 

0.136281 

0.129919 

0.124007 

0.134051 

0.123311 

0.131839 

0.010177 

0.007353 

0.005306 

0.000943 

0.005684 

0.006284 

0.008509 

0.005489 

0.006474 

0.026449 

0.001637 

0.000943 

0.028138 

0.031602 

0.010665 

0.00292 

0.012564 

0.000972 

0.018695 

0.017314 

0.005015 

0.031482 

0.016923 

0.019943 

0.015101 

0.019873 

0.005797 

0.04733 

0.039994 

0.052332 

0.036361 

0.050039 

0.043256 

0.047909 

0.045848 

0.047276 

0.081567 

0.063249 

0.039897 

0.096925 

0.061105 

0.042523 

0.073167 

0.071794 

0.040397 

0.104819 

0.100703 

0.089773 

0.056192 

0.12046 

0.075906 

0.102036 

0.096317 

0.075653 

0.019362 

0.049804 

0.030102 

0.046599 

0.0296 

0.043372 

0.023172 

0.048453 

0.044149 

0.140716 

0.126226 

0.160862 

0.13459 

0.135294 

0.140036 

0.149128 

0.136717 

0.149793 

0.071896 

0.061717 

0.097297 

0.138185 

0.100464 

0.138785 

0.139378 

0.179199 

0.188947 

0.074922 

0.074693 

0.057926 

0.024626 

0.091754 

0.047378 

0.071268 

0.067447 

0.045497 

0.028932 

0.023226 

0.031361 

0.024054 

0.035759 

0.024212 

0.030045 

0.027434 

0.027474 
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936 

1096 

996 
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0.356895 

0.205745 

0.422152 

0.513972 

0.31744 

0.348248 

0.43196 

0.35951 

0.44016 

0.266071 

0.272119 

0.278062 

0.246219 

0.228635 

0.283254 

0.28263 

0.224912 

0.275172 

0.46854 

0.438589 

0.419763 

0.458898 

0.47391 

0.424971 

0.434229 

0.475176 

0.412174 

0.269549 

0.287717 

0.304233 

0.294883 

0.294226 

0.292018 

0.283211 

0.300982 

0.31004 

0.518943 

0.586942 

0.639572 

0.508152 

0.551269 

0.558088 

0.506444 

0.539165 

0.575721 

0.476025 

0.409091 

0.360493 

0.491848 

0.45002 

0.43606 

0.48911 

0.460097 

0.421303 

0.337489 

0.353859 

0.463246 

0.353583 

0.329857 

0.423239 

0.371892 

0.367148 

0.46617 

0.21788 

0.151103 

0.150943 

0.211212 

0.220467 

0.189232 

0.17067 

0.175816 

0.145741 

0.174087 

0.14421 

0.095047 

0.211212 

0.128939 

0.10006 

0.168499 

0.151088 

0.09801 

0.264905 

0.345296 

0.28502 

0.223994 

0.316954 

0.281482 

0.288325 

0.303368 

0.290726 

0.618145 

0.659304 

0.706111 

0.834917 

0.570425 

0.756083 

0.642357 

0.684211 

0.764346 

0.387353 

0.341907 

0.293492 

0.165083 

0.435144 

0.245017 

0.358096 

0.316105 

0.235576 

0.435788 

0.430903 

0.562621 

0.588985 

0.477552 

0.44273 

0.530714 

0.581892 

0.528393 

0.563719 

0.568885 

0.437539 

0.411015 

0.522047 

0.560484 

0.470076 

0.419204 

0.476571 

0.351019 

0.41705 

0.185391 

0.449382 

0.448667 

0.394437 

0.366602 

0.421725 

0.454493 

0.644572 

0.579739 

0.808507 

0.550618 

0.549162 

0.604561 

0.631105 

0.576205 

0.544385 

0.190971 

0.276904 

0.26676 

0.160099 

0.168322 

0.311679 

0.170346 

0.292339 

0.31448 

0.80876 

0.726863 

0.73325 

0.839901 

0.831097 

0.689444 

0.830106 

0.706814 

0.68615 

0.099691 

0.110947 

0.083741 

0.009732 

0.054399 

0.078139 

0.114816 

0.065701 

0.107779 

0.900718 

0.887871 

0.916563 

0.990268 

0.944452 

0.922145 

0.885172 

0.937643 

0.897648 

0.844505 

0.664956 

0.750357 

0.546662 

0.801959 

0.634553 

0.814995 

0.668154 

0.631341 

0.155997 

0.328864 

0.251606 

0.453338 

0.197063 

0.36258 

0.185084 

0.336121 

0.368432 

0.844534 

0.66542 

0.748815 

0.546662 

0.806116 

0.634251 

0.819732 

0.663884 

0.632099 

0.431494 

0.325352 

0.452917 

0.561266 

0.598592 

0.628585 

0.766137 

0.870175 

0.797321 

0.717572 

0.75929 

0.648738 

0.505882 

0.716655 

0.662176 

0.702724 

0.708914 

0.622924 

0.277096 

0.236099 

0.351225 

0.494118 

0.279302 

0.338397 

0.296249 

0.288353 

0.376164 

0.232714 

0.18387 

0.277304 

0.440241 

0.27832 

0.215041 

0.287193 

0.358309 

0.273186 

0.441922 

0.341064 

0.496872 

0.672484 

0.557795 

0.392208 

0.558409 

0.664977 

0.493592 

0.287646 

0.027825 

0.016241 

0.293159 

0.320255 

0.143813 

0.044506 

0.160388 

0.018024 

0.514732 

0.679703 

0.901124 

0.378837 

0.50709 

0.583303 

0.730314 

0.585287 

0.876734 

0.203318 

0.294247 

0.086352 

0.328005 

0.171501 

0.268931 

0.230204 

0.25369 

0.107501 
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0.05469 

0.085792 

0.060641 

0.060345 

0.078525 

0.10187 

0.063556 

0.058898 

0.06788 

0.015037 

0.020611 

0.020939 

0.020999 

0.018465 

0.020326 

0.015093 

0.019543 

0.019296 

0.027913 

0.035662 

0.035364 

0.038183 

0.03274 

0.036204 

0.028225 

0.035767 

0.036536 

0.039473 

0.058584 

0.045971 

0.037216 

0.043354 

0.037298 

0.039696 

0.044502 

0.042338 

0.038899 

0.076942 

0.051778 

0.044345 

0.063274 

0.091304 

0.093965 

0.047275 

0.054229 

0.103389 

0.066388 

0.059129 

0.026699 

0.036644 

0.053627 

0.070233 

0.120157 

0.065886 

0.101412 

0.125418 

0.129194 

0.13186 

0.121651 

0.124603 

0.102899 

0.126682 

0.129717 

0.00539 

0.004833 

0.007367 

0.002165 

0.007452 

0.004583 

0.002807 

0.01066 

0.005045 

0.006259 

0.000371 

0.027897 

0.024617 

0.00731 

0.000437 

0.002608 

0.005059 

0.005736 

0.016365 

0.000176 

0.0237 

0.017659 

0.016817 

0.000491 

0.001715 

0.002157 

0.011848 

0.036065 

0.041648 

0.040555 

0.0524 

0.042674 

0.04436 

0.037115 

0.041615 

0.04678 

0.054934 

0.022796 

0.098838 

0.078137 

0.074009 

0.028895 

0.031431 

0.056456 

0.040124 

0.137508 

0.101634 

0.068823 

0.119461 

0.077261 

0.083343 

0.094053 

0.114939 

0.060097 

0.066428 

0.041721 

0.031045 

0.021592 

0.062612 

0.055401 

0.074739 

0.026689 

0.045375 

0.112607 

0.145811 

0.144805 

0.149488 

0.134866 

0.142633 

0.114688 

0.138791 

0.146144 

0.058859 

0.0995 

0.167912 

0.09925 

0.12013 

0.095455 

0.085344 

0.108036 

0.080588 

0.114002 

0.072292 

0.038553 

0.090284 

0.049237 

0.054636 

0.070341 

0.086198 

0.031574 

0.020953 

0.023047 

0.022176 

0.039784 

0.02651 

0.023611 

0.021957 

0.023174 

0.03038 
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0.153282 

0.290708 

0.473221 

0.35014 

0.356209 

0.342026 

0.277493 

0.22037 

0.312693 

0.165803 

0.292806 

0.242118 

0.22315 

0.232929 

0.284553 

0.284197 

0.213594 

0.274595 

0.530687 

0.402662 

0.486799 

0.472991 

0.446395 

0.395752 

0.420096 

0.507632 

0.411303 

0.301445 

0.302198 

0.270738 

0.308213 

0.318757 

0.314262 

0.299634 

0.28012 

0.318569 

0.520977 

0.529776 

0.594173 

0.634878 

0.541924 

0.548493 

0.530018 

0.568971 

0.651264 

0.473502 

0.465001 

0.404163 

0.365032 

0.458025 

0.451513 

0.47003 

0.429692 

0.350881 

0.385692 

0.475891 

0.386712 

0.334806 

0.354097 

0.450021 

0.412126 

0.486986 

0.474899 

0.121071 

0.115997 

0.204261 

0.204811 

0.165943 

0.125228 

0.104876 

0.11705 

0.167775 

0.103515 

0.056614 

0.203036 

0.165197 

0.165792 

0.073014 

0.074819 

0.111215 

0.097554 

0.384265 

0.356296 

0.205066 

0.298323 

0.314897 

0.350663 

0.402543 

0.278982 

0.256385 

0.552957 

0.682592 

0.776174 

0.572519 

0.762302 

0.754337 

0.693063 

0.609982 

0.80569 

0.44708 

0.313809 

0.223464 

0.42323 

0.23773 

0.248083 

0.308232 

0.390678 

0.194162 

0.633137 

0.553061 

0.421528 

0.425944 

0.622075 

0.652696 

0.421934 

0.484643 

0.379571 

0.366758 

0.446804 

0.579083 

0.578298 

0.376177 

0.348444 

0.581751 

0.515602 

0.620693 

0.502788 

0.430272 

0.46979 

0.456875 

0.406361 

0.290306 

0.297169 

0.484882 

0.439416 

0.495479 

0.565106 

0.52913 

0.544397 

0.593077 

0.710647 

0.703431 

0.515093 

0.562835 

0.50512 

0.346452 

0.314098 

0.168595 

0.231826 

0.301109 

0.405661 

0.486054 

0.33731 

0.495459 

0.654504 

0.686287 

0.832658 

0.769614 

0.699638 

0.594338 

0.512446 

0.664102 

0.083883 

0.103483 

0.074087 

0.022347 

0.100595 

0.092481 

0.063713 

0.179408 

0.073102 

0.916501 

0.90216 

0.925912 

0.980138 

0.900756 

0.91208 

0.93893 

0.820487 

0.927807 

0.674309 

0.708416 

0.689433 

0.846618 

0.549633 

0.60056 

0.556174 

0.811603 

0.569897 

0.325748 

0.290807 

0.310993 

0.153023 

0.445418 

0.399215 

0.441961 

0.188453 

0.430286 

0.675562 

0.706869 

0.689089 

0.847159 

0.555028 

0.601677 

0.552275 

0.812267 

0.56939 

0.353112 

0.482358 

0.799049 

0.562457 

0.494384 

0.402665 

0.41097 

0.632277 

0.313978 

0.845897 

0.762312 

0.637238 

0.692125 

0.649748 

0.700796 

0.761623 

0.787656 

0.510461 

0.155471 

0.24303 

0.366541 

0.304985 

0.349839 

0.302849 

0.237736 

0.211757 

0.491153 

0.474632 

0.297383 

0.251352 

0.256309 

0.388861 

0.373758 

0.172239 

0.311881 

0.173835 

0.881091 

0.514542 

0.424506 

0.466045 

0.689487 

0.66572 

0.3221 

0.570812 

0.329148 

0.106283 

0.008806 

0.303006 

0.25926 

0.109559 

0.009677 

0.06304 

0.103762 

0.089593 

0.612449 

0.988376 

0.4405 

0.551868 

0.639551 

0.981369 

0.897207 

0.853611 

0.730663 

0.277904 

0.004185 

0.257426 

0.185982 

0.25203 

0.010854 

0.041456 

0.044243 

0.18505 
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0.042497 

0.062336 

0.054755 

0.078346 

0.084789 

0.075029 

0.052752 

0.114758 

0.099455 

0.018895 

0.022409 

0.019745 

0.018231 

0.021048 

0.016716 

0.015968 

0.017747 

0.019541 

0.033833 

0.038894 

0.036674 

0.032256 

0.038887 

0.030658 

0.032693 

0.033831 

0.035999 

0.046133 

0.052131 

0.014201 

0.047906 

0.048406 

0.04938 

0.026584 

0.023857 

0.048666 

0.047421 

0.063719 

0.092151 

0.064365 

0.062857 

0.06969 

0.096475 

0.047508 

0.068146 

0.019624 

0.052631 

0.031553 

0.052168 

0.063475 

0.081708 

0.056755 

0.086944 

0.075207 

0.12319 

0.149535 

0.130841 

0.122356 

0.139522 

0.113221 

0.117876 

0.122864 

0.127816 

0.00606 

0.00291 

0.005774 

0.011665 

0.007306 

0.007452 

0.006748 

0.008743 

0.004708 

0.037546 

0.009865 

0.036099 

0.000887 

0.010207 

0.00098 

0.033869 

0.039579 

0.001231 

0.020778 

0.001641 

0.024413 

0.000157 

0.021235 

0.012489 

0.01665 

0.025748 

0.001564 

0.044738 

0.059204 

0.048761 

0.039851 

0.048369 

0.038792 

0.044192 

0.043304 

0.044794 

0.107278 

0.056996 

0.10527 

0.047176 

0.081433 

0.058737 

0.053019 

0.031889 

0.016472 

0.07707 

0.082747 

0.082782 

0.109378 

0.06484 

0.122209 

0.085916 

0.083357 

0.06511 

0.032331 

0.029088 

0.034374 

0.04677 

0.022668 

0.038623 

0.043309 

0.038595 

0.044543 

0.136949 

0.160335 

0.143833 

0.139196 

0.156558 

0.119763 

0.128041 

0.135543 

0.146683 

0.11586 

0.071046 

0.054102 

0.082765 

0.136067 

0.070954 

0.094099 

0.119076 

0.036252 

0.0481 

0.053321 

0.051482 

0.081126 

0.032303 

0.095816 

0.059588 

0.054387 

0.036813 

0.0275 

0.046411 

0.028177 

0.019489 

0.026957 

0.018489 

0.03048 

0.024396 

0.025399 
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0.460442 

0.412372 

0.451144 

0.244437 

0.467743 

0.166317 

0.346575 

0.324845 

0.328599 

0.259642 

0.231532 

0.212774 

0.267451 

0.264699 

0.251858 

0.257498 

0.239576 

0.309418 

0.470447 

0.446504 

0.501968 

0.432861 

0.461765 

0.47421 

0.461083 

0.484255 

0.380507 

0.269543 

0.316191 

0.282883 

0.299603 

0.27055 

0.271249 

0.282286 

0.275869 

0.31006 

0.634058 

0.499999 

0.500079 

0.579124 

0.571884 

0.526901 

0.524035 

0.698808 

0.621505 

0.360797 

0.499958 

0.499958 

0.420958 

0.421901 

0.473277 

0.476721 

0.298377 

0.377611 

0.303121 

0.452829 

0.323665 

0.403434 

0.439521 

0.411117 

0.378857 

0.433004 

0.533001 

0.232033 

0.163425 

0.228797 

0.121545 

0.188547 

0.125772 

0.220891 

0.243441 

0.136799 

0.228033 

0.12765 

0.209715 

0.108987 

0.176352 

0.123863 

0.114989 

0.065851 

0.04329 

0.232099 

0.250577 

0.233423 

0.360398 

0.189693 

0.33954 

0.27951 

0.252414 

0.288175 

0.719528 

0.683527 

0.720323 

0.662599 

0.781432 

0.580681 

0.683085 

0.714875 

0.799232 

0.280473 

0.316433 

0.281364 

0.336216 

0.219072 

0.419404 

0.320404 

0.28558 

0.200731 

0.258138 

0.538464 

0.514593 

0.505825 

0.56012 

0.565382 

0.390932 

0.68546 

0.517177 

0.745929 

0.466283 

0.485208 

0.492344 

0.440421 

0.434609 

0.603677 

0.314496 

0.48355 

0.492467 

0.450355 

0.133499 

0.425505 

0.433444 

0.414266 

0.216561 

0.333739 

0.416523 

0.506217 

0.550462 

0.866255 

0.5717 

0.562846 

0.584657 

0.785917 

0.664596 

0.583248 

0.13761 

0.260305 

0.194212 

0.298733 

0.312753 

0.419111 

0.3249 

0.414641 

0.370824 

0.863875 

0.739574 

0.805329 

0.700659 

0.687451 

0.580751 

0.674792 

0.585947 

0.630224 

0.055518 

0.039884 

0.050279 

0.221  72 

0.083914 

0.12494 

0.066254 

0.074163 

0.090449 

0.944541 

0.965433 

0.949736 

0.775651 

0.916272 

0.875623 

0.933893 

0.925862 

0.913086 

0.705829 

0.739577 

0.706506 

0.70113 

0.740242 

0.758998 

0.666649 

0.681955 

0.593787 

0.2961 

0.25998 

0.29337 

0.299806 

0.258785 

0.239876 

0.336052 

0.315749 

0.40622 

0.706128 

0.741864 

0.705868 

0.70118 

0.74052 

0.760358 

0.664028 

0.687297 

0.591918 

0.597389 

0.328724 

0.265509 

0.416915 

0.643597 

0.450478 

0.488 

0.6038 

0.146289 

0.63624 

0.532928 

0.64681 

0.805365 

0.545018 

0.842239 

0.658826 

0.690345 

0.591487 

0.36376 

0.463868 

0.354018 

0.193475 

0.454817 

0.162519 

0.337001 

0.309663 

0.40809 

0.153869 

0.415144 

0.382449 

0.239075 

0.315706 

0.289834 

0.196024 

0.337054 

0.210144 

0.275514 

0.720528 

0.710345 

0.422983 

0.583281 

0.531565 

0.401  342 

0.642531 

0.387127 

0.364155 

0.140034 

0.330948 

0.021738 

0.127947 

0.018768 

0.356078 

0.36262 

0.025899 

0.433908 

0.840393 

0.447038 

0.976524 

0.606319 

0.742795 

0.46461 

0.396749 

0.942468 

0.201518 

0.02329 

0.223812 

0.003853 

0.266189 
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